MarkLogic and NoSQL

I’ve been itching to write up a post about the NoSQL (“not only SQL”) category of technologies because there’s such a dearth of practical and specific information on this topic, and because so many people are unclear about how MarkLogic compares to these technologies.

This post is targeted at folks who have already come to the realization that a relational database won’t meet their needs, and who are trying to figure out which of the available alternatives is a plausible option. It’s also meant to be a general tutorial for anyone curious about this emerging space.

The NoSQL “movement” was born from a growing recognition that there are certain types of data-based problems that are quite difficult, or inefficient, or impossible-within-practical-constraints to tackle using relational database technology. The most common factors behind these problems are unstructured data and scalability. Why?

• Unstructured data doesn’t fit neatly into rows and columns.
  • Expensive and time-consuming efforts are required for data modeling activities aimed at mapping pieces of unstructured data to table cells. Inevitably the process results in some amount of fidelity loss.
  • Business impedance results from being tied to a rigid schema based architecture, because unstructured data evolves quickly.
• Relational databases scale vertically, not horizontally.
  • There gets to be a point where even the most exotic hardware can’t handle the complexity and/or load of these types of problems.
  • Today’s data centers and cloud vendors are built around scaling out on commodity hardware.
  • There is no efficient way to distribute RDBMS managed data across a growing (or shrinking) cluster of servers.

The NoSQL technologies (I don’t include MarkLogic in this category) are not databases “in the traditional sense”, meaning none of them provide both transactional integrity and real-time results, and some of them provide neither.

Each resulted from an effort to alleviate specific limitations found in the RDBMS world which were preventing their architects from completing a specific type of task, and they all made trade-offs to get there. Whether it was tackling “hot row” lock contention, horizontal scale, sparse data performance problems, or single schema induced rigidity, they are much more narrowly focused in purpose than their RDBMS predecessors. They weren’t designed to be enterprise platforms for building ACID transactional, real-time applications.

Data consistency is the most prominent NoSQL trade-off made in order to achieve horizontal scale, so it’s worth understanding generally why and generally what the ramifications are. There is an obvious data consistency challenge with cache-based NoSQL options (such as memcached) so I’ll focus on the issues with persistent stores here.

The concept is relatively straightforward–figure out how to partition your data such that it will be most evenly distributed across a cluster of commodity servers. This requires some upfront understanding about what kinds of questions you’ll be asking, such that all of the results aren’t clumped together within the cluster. (If you simply rely on the default consistent hash algorithm provided by some of the NoSQL technologies, you’ll not be fully optimized because that technique is not content-aware). If the questions change, repartitioning might be needed to maintain performance. In order to avoid “hot spots”, (which introduce lock-contention based performance problems), multiple copies of the data exist within the cluster. With some NoSQL options, joins are not possible, so data cannot be normalized, and again you end up with multiple copies of data. Either way, you have a data consistency issue, because when updates occur, for some window of time some of the nodes in the cluster will have stale data. This is known as “eventual consistency”, a model used by Amazon Dynamo and copied by many. In short, updates are treated as “always” writeable, with the complexity of potential conflict resolution left to the read operation. None of the NoSQL options have anything other than rudimentary conflict resolution functionality, so this is left as an exercise for the application/business-process layers.

Obviously, I cannot provide exhaustive descriptions of each NoSQL subcategory here, but since many people expect NoSQL options to be “databases”, I’ve tried to highlight those characteristics that might be least expected.

Key-Value Store

• Essentially a distributed hashmap for simple store and retrieve only
• Achieves flexibility by not requiring a single schema; achieves horizontal scalability through simplified key-based access, key-based partitioning, and eventual consistency
• No joins, no search, no global indexes, no complex queries
• Can be in-memory or persistent
• Examples: memcached, Amazon SimpleDB, Tokyo Tyrant, Redis, Berkeley DB, Voldemort, Oracle Coherence, Gigastore

Document Store

• Achieves flexibility by not requiring a single schema; addresses sparse data problems by replacing “rows” with “documents”, where documents can have embedded documents, allowing for complex hierarchies to be stored in a single record; achieves horizontal scalability through key-based partitioning and eventual consistency
• Allows for more complex queries than key-value stores
• No joins
• No complex multi-document transactions
• Weak full text search functionality. All keywords need to be placed in a single field. The application code needs to handle tokenization and stemming.
• You need to configure indexes explicitly, and there are limitations on how many indexes you can specify. (e.g. in MongoDB you are limited to 64 indexes per collection). It’s recommended to configure the minimum number of indexes possible, in order to preserve performance.
• Examples: MongoDB, CouchDB

Map Reduce Batch Processing

• It’s a framework for distributed computing on large datasets, not a database.
• Unlike the other noSQL technologies, map reduce (in combination with a distributed filesystem) is fundamentally targeted at addressing the limitation of disk read performance rather than RDBMS shortcomings.
• Good for writing once, reading many times.
• Good for offline batch processing—results come back in minutes, hours or days, not instantly.
• Relies on dedicated hardware in a single data center with very high aggregate bandwidth interconnects.
• Data is partitioned based on the basic questions you are asking
• No search or retrieval
• No global indexes
• No transactions
• Examples: Hadoop and HDFS

Columnar Store

• Multi-dimensional key-value store which provides richer data model than straight key-value pairs via facility to organize columns
• Addresses sparse data problems by storing data based on columns instead of rows; achieves horizontal scalability through master/chunk-server partitioning (with the exception of Cassandra which follows the Amazon Dynamo model, ie eventual consistency)
• Well-suited for computing aggregates over very large sets of similar data
• Often require data to be of uniform type
• Examples: Cassandra, HBase, Google BigTable, HyperTable