09. Sep 2012 01:09
No comments

NoSQL on the spot

Huge letter blocks saying
Huge letter blocks saying

With the vast amount of NoSQL solutions out there it requires huge effort to make an informed choice. Though one does not in fact need to set up every single NoSQL solution to make such a choice. In many cases usage pattern can be summarized into a set of solid technical requirements. With such a set one can now simply rule out solutions by their features. We found that in fact the feature set alters greatly between different solutions. Here we will go through a set of requirements and show each solution's take on it.

Effortless thorough feature comparison

We will use a technique that allows us to gather referenced data to inform a greater amount of people within the stake-holder groups. It may sound simple but gives a huge benefit when it comes to arguing for a particular solution. As afore mentioned we will guide through our search with a set of requirements, where we want to make sure that most can be answered with a quick "YES it has this feature" or "NO it doesn't have this feature." Whilst this means we provide quick answers for the impatient, we have to make sure to be able clarify on both answers. The easiest way to do so is simply taking a quote of the exact part of the documentation that backs your claim (i.e. theYES or NO) and a link to the page it comes from. This way, if in doubt about a certain claim one can simply read the quote and if in doubt whether it is up to date, the link to its origin allows quick refreshing of the information. Some of the chosen criteria can be a non-"YES or NO" criteria to allow further summarizing of knowledge. However to keep the comparison transparent, we should try and focus our decision making on the existence or non-existence of required features, which may mean a criteria has to be further specialized to become a "yes or no"-criteria.

Using such a simple technique will quickly rule out a vast amount of solutions as one can make strong statements on a particular requirement and immediately understand how a system fits into the set of requirements and what its down-sides are. Keeping such comparison version controlled over time also means that one will quickly understand when a chosen solution is coming of age and can react timely.

Comparison Criteria

As mentioned this is  something that may greatly differ, depending on your goals. Other than usual where we pick simple examples to ease understanding, in this article we want to pick a set of criteria that actually fulfills a wide range of systems and be as critical as possible. Following are our criteria with a short description what we want from each criteria.

Internal partitioning - The system is capable of partitioning / sharding data across multiple nodes to be able to distribute data for capacity or ideally even performance optimization. As part of the explanation to this criteria we want a brief sum up of how data is partitioned.

Automated flexible data distribution - Being able to partition data alone is not enough, the data has to be rearranged by the system itself, manual interference to move partitions around is undesirable as a requirement for human reaction will cause long lasting outages of subsets of the stored data. 

Hot swappable nodes - The system needs to be able to populate newly added nodes and reinstate nodes that have failed. Reinstatement does not mean the same node being put back in place, but rather for the data to become available again or better still, stay available throughout the failure.

Replication-style - This is not a yes no criteria as we are interested in gathering the style of replication, i.e Master to Master, Master to slave and other informations that may be of interest to our problem.

Automated failover strategy - Seemingly not a yes or no criteria but in fact it is, here we inquire a solutions mechanism to fail over traffic to a separate source may it be a hot failover node or instatement of a cold backup. Any manual interference required during a fail over would mean this criteria is not satisfied.

Those are our five criteria, many of which play into each other and you may argue if you have a flexible data distribution you of course have hot swappable node and that may well be the case but listing both individual allows us to gather data on how either mechanism is implemented and thus may allows us to rule out a specific kind of implementation of those features

Comparison of NoSQL solutions by five core criteria

From now on we will continue with the actual comparison, following the same style as above. Though the descriptions following each criteria will be a quote of the system's documentation with a link referencing its origin. As seen in the example below:

Replication-style - No"Replication? what's replication? our nodes never fail!"

All of the statements are taken from public pages and are part of each of the system's documentation, some further knowledge on some of these systems can be taken from their academic publishing, such should however not flow into this document as not all of these documents are freely available.

The list is done in no particular order and it is clear that some systems simply weren't made to fulfill the listed criteria and are only here to clarify exactly that.


Internal partitioning -Yes - "MongoDB scales horizontally via an auto-sharding (partitioning) architecture."

Automated flexible data distribution Yes - "From times to time, the balancer may decide to move chunks around. It is possible to issue a manual command" 

Hot swappable nodes Yes - "Adding a new node to an existing replica set is easy."
- http://docs.mongodb.org/manual/tutorial/expand-replica-set/

Replication-style "MongoDB supports asynchronous replication of data between servers for failover and redundancy. Only one server (in the set/shard) is active for writes (the primary, or master) at a given time."

Automated failover strategy Yes - "Each shard will consist of a group of n servers in a configuration known as a replica set. If any one server in the replica set fails, read and write operations on the shard are still permitted. What's more, no data need be lost on the failure of a server because the replica allows an option on write that forces replication of the write before returning."


Internal partitioning - Yes - "Each Cassandra server node is assigned a unique Token that determines what keys it is the first replica for. If you sort all nodes' Tokens, the Range of keys each is responsible for is (PreviousToken, MyToken), that is, from the previous token (exclusive) to the node's token (inclusive)." 

Automated flexible data distribution Yes but manual - "If you add nodes to your cluster your ring will be unbalanced and only way to get perfect balance is to compute new tokens for every node and assign them to each node manually by using nodetool move command."

Hot swappable nodes Yes - "Read and write throughput both increase linearly as new machines are added, with no downtime or interruption to applications."

Replication-style "Data is automatically replicated to multiple nodes for fault-tolerance. Replication across multiple data centers is supported. Failed nodes can be replaced with no downtime."

Automated failover strategy - Yes - "Data is automatically replicated to multiple nodes for fault-tolerance. [...] Failed nodes can be replaced with no downtime." 


Internal partitioning - Yes"Riak computes a 160-bit binary hash of the bucket/key pair, and maps this value to a position on an ordered "ring" of all such values. This ring is divided into partitions. Each Riak vnode is responsible for a partition (we say that it "claims" that partition)."

Automated flexible data distribution - Yes - "Riak is built so you can add more capacity as your app or platform grows. When you add new machines, Riak distributes data automatically around the cluster with no downtime and a near-linear increase in performance and throughput."

Hot swappable nodes - Yes - "The ring state is shared around the cluster by means of a "gossip protocol". Whenever a node changes its claim on the ring, it announces its change via this protocol. It also periodically re-announces what it knows about the ring, in case any nodes missed previous updates." 

Replication-style - "Replication is fundamental and automatic in Riak, providing security that your data will still be there if a node in your Riak cluster goes down. All data stored in Riak will be replicated to a number of nodes in the cluster according to the n_val property set on the bucket."

Automated failover strategy - Yes"Writing And Reading When One Primary Failed and Later Recovered: One primary goes down; Data is written with W=3; A secondary takes responsibility for the write; The primary comes back up; Within a default timeframe of 60 seconds, hinted handoff will occur,transferring the updated data to the primary; After handoff occurred, the node can successfully serve requests" 
- http://wiki.basho.com/Eventual-Consistency.html#Failure-Scenarios  

CouchBase 2.0

Internal partitioning - Yes - "A bucket is a logical grouping of physical resources within a cluster of Couchbase Servers. They can be used by multiple client applications across a cluster. Buckets provide a secure mechanism for organizing, managing, and analyzing data storage resources."

Automated flexible data distribution - Yes - "These vBuckets are used to allow information to be distributed effectively across the cluster. The vBucket system is used both for distributing data, and for supporting replicas (copies of bucket data) on more than one node. vBuckets are not a user-accessible component, but they are a critical component of Couchbase Server and are vital to the availability support and the elastic nature." 

Hot swappable nodes - Yes - "Couchbase Server is designed to actively change the number of nodes configured within the cluster to cope with these requirements, all while the cluster is up and running and servicing application requests."

Replication-style - "You can use XDCR [Cross DataCentre Replication] to provide a live backup of your application data in a separate cluster, either locally or within a different datacenter, geographically local or distant from your live running cluster. In the event of a system failure at your main site, you can either enable the secondary cluster, or use the data stored in the secondary cluster to re-populate the data in the primary cluster."

Automated failover strategy - Yes - "Failover can be performed manually, or you can use the built-in automatic failover that reacts after a preset time when a node within the cluster becomes unavailable."

HBase distributed on HDFS using ZooKeeper

Internal partitioning - Yes"Regions are the basic element of availability and distribution for tables [...] HBase scales by having regions across many servers." 

Automated flexible data distribution - Yes - "The HDFS architecture is compatible with data rebalancing schemes. A scheme might automatically move data from one DataNode to another if the free space on a DataNode falls below a certain threshold. In the event of a sudden high demand for a particular file, a scheme might dynamically create additional replicas and rebalance other data in the cluster. These types of data rebalancing schemes are not yet implemented."

Hot swappable nodes - Yes"In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an Active state, and the other is in a Standby state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary."

Replication-style - "The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later."

Automated failover strategy - Yes"Each DataNode sends a Heartbeat message to the NameNode periodically. A network partition can cause a subset of DataNodes to lose connectivity with the NameNode. The NameNode detects this condition by the absence of a Heartbeat message. The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them."

Berkeley db 11g (java Ed HA) 

Internal partitioning No - "Berkeley DB's replication doesn't partition the data, every node keeps an entire copy of the database." 

Automated flexible data distribution Yes - "Configurable background cleaner threads re-organize data and optimize disk use" 

Hot swappable nodes Yes - "new replicas can join the group at any time to scale the system" 

Replication-style - "all updates go to a designated master, which distributes changes automatically to a set of replicas"

Automated failover strategy - Yes - "The master-failover process generally takes only a fraction of a second and read-requests can be serviced by replicas during that fail-over period ensuring zero downtime." 

Oracle NoSQL 11g

Internal partitioning Yes - "Automatic, hash-function based data partitioning and distribution; Intelligent NoSQL Database driver is topology and latency aware, providing optimal data access" 

Automated flexible data distribution No but manual - "it is possible to simply run a ‘Storage Migration Plan’ to relocate the StorageNode to a different physical system."

Hot swappable nodes Yes but manual - "If a Storage Node has failed, or is in the process of failing, you can replace the Storage Node. [...] On the new, replacement node, create a "boot config" configuration file using themakebootconfig utility. [...]" 

Replication-style "Oracle NoSQL Database provides a single- master, multireplica highly available database replication. Transactional data is delivered to all replica nodes with flexible consistency policies per transaction." 

Automated failover strategy Yes - "In the event the master replica node fails, a PAXOS-based automated fail-over election process minimizes downtime." 


Our criteria is widely supported across the compared systems though implemented in a wide range of fashions. Some solutions stand out due to their complexity and others take inspiration from systems otherwise not available to the wider public, such as the GoogleFileSystem and Dynamo. We have only compared solutions that can actually be used by anyone (though some may incur costs). 

Choosing the right solution always depends on your requirements and our listing has purely concentrated on the different distribution models and left any aggregation features and the complexity of such aside.

Books about the topic „NoSQL“

The following books are all about the topic "NoSQL" and are highly recommended. I have not read all of these books, but a good number of them and some of them I used for my research as well.

Comments about the topic „NoSQL on the spot“

If you like you can leave a comment about the topic and exchange with other reads. In order to comment you need to login and then you can start immediately.
Login now to comment