Trifork Blog

Category ‘NoSQL’

City-wide crowd management in Amsterdam

November 10th, 2015 by
(http://blog.trifork.com/2015/11/10/city-wide-crowd-management-in-amsterdam/)

As most residents and visitors of Amsterdam know, every year more people are visiting Amsterdam, city wide events like GayPride, Koningsdag and MuseumNacht are getting bigger and more frequent, putting more strain on the city’s infrastructure and all people living in the city center.

That’s why this November 7th, Amsterdam Marketing organized the Museumn8 hackathon to allow developers to come up with creative and innovative solutions for improving improving mobility, navigation and crowd management in the city. Twenty teams eventually participated.

Trifork (Rienk Prinsen, Marleine van Kampen, Marijn van Zelst) and weCity (David Kat, Luc Deliance) teamed up and joined the hackathon to give their take on solving this problem. Their solution:

city-live-logo

By transforming the advertisement billboards of JCDecaux into large information screens displaying real-time information, visitors can get informed about activities and interesting places in the vicinity of the billboard. They receive live crowd information, travel times to and queue lengths at museums and even recommendations where to go next. Read the rest of this entry »

Efficiently storing your domain model in Riak

June 20th, 2013 by
(http://blog.trifork.com/2013/06/20/efficiently-storing-your-domain-model-in-riak/)

Riak_product_logoDomain modelling and persistence appear to be at odds with each other, no matter which persistence store you use. Over the years practices have been developed for storing a model in a relational database using ORM frameworks like Hibernate, and various design patterns to help mitigate a number of issues. Not all of these translate very well when using one of the NoSQL persistence stores however. In this blog I will describe a situation we recently came across when working with a model we have to store in Riak. I will detail the model, the typical relational solution in Riak and what alternatives you may want to consider to better work with the strengths and weaknesses of Riak.

Read the rest of this entry »

Eventual Consistency

May 16th, 2013 by
(http://blog.trifork.com/2013/05/16/eventual-consistency/)

Once in a while, an idea emerges that is contrary to the way we have grown accustomed to doing things. Eventual consistency is such an idea, and the way we used to build datastores was with SQL and ACID transactions.

So whats wrong with that?

Too many generals

Information always flows as messages using a medium – not transactions. Atomic transactions spanning multiple systems is an abstraction that doesn’t really exist in real life. It’s impossible to guarantee the atomicity of a distributed transaction as proved by the Two_Generals%27_Problem. The more generals – or partitions of a distributed system – the worse the problem. Not knowing the entire truth, is thus a fact of life that each partition must deal with, and therefore we need to ensure that the knowledge of all partitions converges when they send messages to eachother. This is eventual consistency.

Eventual Consistency in Real Life

Finance has always embraced eventual consistency. Money transfer is – contrary to popular belief – not done by transactions, but is an elaborate process where the money is first withdrawn from one account and after some time deposited at another account. Meanwhile the system is inconsistent for the outside observer, due to money being “in movement”. When the money arrives, eventually, the system will again be consistent.

If I send a cheque by mail when I have enough money on my account, but it turns out I have insufficient funds, when the receiver tries to cash it, a conflict arises due to balance information being slow and inconsistent. The entire system still manages because we have processes for resolving conflicts. In this case the cheque bounces.

It turns out, that in all cases of distributed updates to an object predating computerisation, eventual consistency is the norm, and conflict resolution processes exist. Medical records is another example where information from many sources about the same patient arrive out of order and is eventually reconciled.

Early computerisation changed this by building monolithic systems to model the truth about something – and when the system was not available, users had to wait or go back to the old and robust processes without computers.

Physical limitations

Consider the physics of sending information. Disregarding exotic theories involving multiple timelines, information cannot move faster than light. This would lead to a paradox, where you could make a phone call into the past with a so-called Tachyonic_antitelephone.

In the globally connected world, location and travel of information imposes very real limitations on the availability of up-to-date data. Connections drop, systems go down, sometimes entire data centers go offline for extended periods of times, and come back up using a restored, old version of data. When building highly distributed systems, these limitations cannot be sanely abstracted away in underlying distributed databases. The ugly head of reality will either cause distributed transactions to crumble write availability, or make cracks in the flawed illusion that all users always see up-to-date data.

Dealing with eventual consistency

It is the gut instinct of every programmer to avoid complexity by abstraction. Since the old abstractions of transactions and a perfectly up-to-date database don’t scale to distributed systems with high write availability, we need eventual consistency as theoretical basis and new abstractions and techniques for handling it. Luckily this has been the subject of academia for decades, and recently the NoSQL movement has pioneered numerous small and large-scale eventually consistency systems drawing many real-life experiences, which will be the topic of my next blog entry.

Rune Skou Larsen
NoSQL evangelist @Trifork

10,9,8, the countdown to Mars has started…

April 22nd, 2013 by
(http://blog.trifork.com/2013/04/22/1098-the-countdown-to-mars-has-started/)

logo_marsoneA big congratulations to one of our customers Mars One who this afternoon announced the start of their astronaut selection program at a press conference in New York.

For those of you who have not heard heard of them, Mars One is a not-for-profit organization that will establish a permanent human settlement on Mars in 2023 through the integration of existing, readily available technologies from industry leaders world-wide. Unique in its approach, Mars One intends to fund this decade-long endeavor by involving the whole world as the audience of an interactive, televised broadcast of every aspect of this mission, from launch to landing to living on Mars.

Over the last few months Trifork has been working together with the team at Mars One to build a global platform that will support Astronaut Selection Program. A scalable site has been developed which expects applications from hundreds, perhaps thousands more likely even millions of applicants in the coming months from all across the world.

Using the services from SoftLayer, MongoDB, Bits on the Run and a lot of other cool technologies and services, we’ve been able to create a very robust website which will make everyone across the world  able to participate in this great journey.

You can watch the press conference on Youtube: http://www.youtube.com/watch?v=WJNGH4NZJ4U

Putting the pedal to the Mongo metal

March 1st, 2013 by
(http://blog.trifork.com/2013/03/01/putting-the-pedal-to-the-mongo-metal/)

At Trifork we’re always looking to get the best performance out of our systems. As a 10gen partner that means that we also try to squeeze the most out of our MongoDB deployments in terms of read and write throughput. Experience has shown that it matters greatly whether those deployments are performed on dedicated hardware or on virtual machines: especially having enough and fast RAM and disk IO can make all the difference.

If you’re interested in reading up on this topic, make sure to check out this article on SoftLayer’s blog. It shows how hosted MongoDB deployments can perform when given the appropriate hardware, and how having a hosting provider that understands the technology running on their infrastructure can help you to achieve the performance that your applications require.

If you’re in The Netherlands and want to learn more about MongoDB, whether you’re a newbee or an experienced user already, you should sign up for  the upcoming Mongo User Group meetup that’s taking place coming Tuesday as well: SoftLayer will be there to tell more about their hosted MongoDB offering, 10gen’s Alvin Richards will talk about what’s planned for the upcoming releases of MongoDB (spoiler: it includes search!) and we’ll have Open Space sessions where you can decide for yourself what topics you’d like to see covered. Hope to see you there!

And that was Devoxx 2012, partnering with 10gen and a lot of knowledge gaining

November 17th, 2012 by
(http://blog.trifork.com/2012/11/17/and-that-was-devoxx-2012-partnering-with-10gen-and-a-lot-of-knowledge-gaining/)

Last week a couple of my colleagues and I were in Antwerp visiting the Devoxx conference. In this blog post we try to give you an idea about what we did & learned:

The sessions

I attended a lot of sessions, if you want to know more I urge you to read on. I briefly discuss the following talks:

  • Keynote: Geek leaks
  • New features in Java 8
  • Security and http headers
  • Developer tools in google chrome
  • OAuth2
  • Skaling software with akka
  • Last but not least: AngularJS

Read the rest of this entry »

Orange11 announces MongoDB partnership with 10gen

May 21st, 2012 by
(http://blog.trifork.com/2012/05/21/orange11-announces-mongodb-partnership-with-10gen/)

At Orange11 we’re very happy to be able to inform you that we’ve recently become an official 10gen MongoDB Service Partner. MongoDB, the leading document-based database solution that allows high-performance and high-scalability data access without many of the limitations imposed by traditional relational database systems.

More on MongoDB
If you haven’t heard of MongoDB yet, then apparently you haven’t been keeping up with the whole NoSQL movement. NoSQL is the term used to label a whole new category of data storage and retrieval solutions that vary widely in their underlying models and offered functionality, but all have in common that they are not based on the relational database paradigm that’s still predominant as the storage solution for enterprise applications. Types of NoSQL databases include key-value stores, column databases, document-oriented databases and graph databases; roughly speaking this list is ordered ascending in terms of the richness of the supported data structures and descending in terms of raw performance and ease of horizontal scalability. In the document-based family of NoSQL solutions, MongoDB has managed to become the nr. 1 solution used by both cutting-edge start-ups as well as big established companies.

That means that MongoDB uses a document that’s part of a collection as the basic unit of storage, instead of a row in a table like with an RDBMS. This document is modeled in BSON, a binary format for representing JSON documents. Documents that you store in any one collection typically share a similar structure, but do not have to conform to an exact schema. That means that it becomes very easy to change or expand the exact contents of your documents. This is a great feature: not only does it enable rapid development in an agile setting, where data model requirements are still fluid, but it also allows working with semi-structured data easily; e.g for storing and querying the results of JSON-based RESTful APIs without having to pre-process your data. Documents can be nested, so you can easily group related data (like an order with its items) in a single document, avoiding the need for what would be a JOIN in a relational database schema.

On the other hand, many tried and proven techniques familiar from the relational database world are offered by MongoDB as well. You can add indices to improve query performance, write operations can be guaranteed to be durable (but don’t have to be), read operations can be fully consistent (but again, don’t have to be if your performance and scalability needs are better served by using eventual consistent reads instead), etc.

Changes to documents can be modeled as atomic unit-of-works and are supported by a find-and-modify style of operations that in many cases prevents the need for techniques like optimistic locking on a whole document level. For more advanced operations you can even write map-reduce jobs that execute directly at the database level.

Working with MongoDB

MongoDB can run on Unix, OS X and Windows. It’s developed as an open source project, which has led to a lot of support for it in many different languages and frameworks. 10gen provides drivers for the major programming languages like Java, but you can find community supported drivers for many more languages. These drivers allow for programming directly to the MongoDB API where JSON-like constructs are mapped to the natively supported data types of your language. If you prefer something more high-level, for example to map Java objects to documents, there are various projects and frameworks providing support for that as well.

If you are a regular reader of this blog, you might be familiar with the Axon CQRS framework that’s developed by Orange11. As part of the work that’s underway for Axon 2.0, MongoDB support is added directly to the framework. This means you will be able to use MongoDB as an event and saga store, and that events can be serialized directly to BSON instead of to XML using XStream to achieve the best possible performance. This work is almost complete: for more details, you can visit these blog entries:

Using the Spring data project and the MongoDB adapter specifically

A MongoDB based Axon framework event store

About the partnership

As a Service Partner, Orange11 provides various services around MongoDB. As a full service solution provider we can make MongoDB is a part of the custom applications we create for organizations through integration with the data access layer, but we can also advice you about the deployment topology to use for your MongoDB servers to ensure high availability and scalability that meets your data demands. Being an official 10gen partner we have direct access to the people developing and supporting the product, thereby providing our customers with the safety of knowing that they’re not only benefiting from Orange11’s expertise but that they’re backed by support straight from the source whenever necessary.

Finally, 10gen is a silver sponsor of this week’s GOTO Amsterdam conference (co-produced by Orange11) and Alvin Richards, Senior Technical Director for 10gen/MongoDB in Europe will be a speaker on Thursday’s program. We have a joint booth at the event, so make sure to drop by if you’re attending to find out more about MongoDB and how Orange11 can help you to get the best results with this innovative new technology!

We are excited about this partnership and look forward to sharing our insights within the community and our customers. Just contact us if you want to know more or take a look at our website.

P.S. Check out the recently created MongoDB User Group for the Netherlands and join us to find out more.