Gartner Cloud DBMS Report Names MarkLogic a Visionary

Scaling Your Database Doesn’t Have to be Hard

A friend of mine tweeted this out, looking for feedback:

Move complex queries out of the database and into service logic because it’s way easier to scale services horizontally?

A couple of the follow-ups showed a pain many of us have felt:

  • Absolutely. Have proven it myself many times. Databases don’t scale.
  • of course databases can scale, but pushing a button in Marathon is easier than sharding strategies

Is it true that databases don’t scale?

Is it easier to scale services than the database? As with many things, I’ll go with “it depends”.

Relational databases are hard to scale. You either spend a lot to scale up, or you deal with sharding as you scale out. With those choices, I can see the appeal of moving complex queries into a service layer. But there’s another way.

MarkLogic stores data as documents. A database is broken down into forests (the physical data storage), which includes the content of documents stored in the forest as well as any index values related to those documents. One of the benefits of this approach relates to how MarkLogic scales out: you don’t have to figure out a sharding strategy.

When you add a new server to a MarkLogic cluster, MarkLogic automatically shifts documents around — transactionally — to ensure an even distribution. MarkLogic Server is designed to scale to 100s of nodes or beyond. What does that mean for queries?

Suppose you have a cluster of five MarkLogic servers, each with about one-fifth of the content. A MarkLogic server that hosts data is known as a data node (d-node); a MarkLogic server that responds to queries is known as an evaluator node (e-node). Commonly, a server will be both an e-node and a d-node. Your query goes to one of the servers, which will act as the e-node for that query. The e-node sends the query to each of the data nodes (d-nodes) in the cluster. Each d-node will use its own indexes to determine which documents that it owns match the query, then returns an identifier and a score for each. The e-node picks the top scores from among the responses, then asks the owning d-nodes for the content. Note that each server in the cluster was able to determine which of its documents matched the query without input from others.

Let’s come back to the original question: should we move complex queries out of the database and into service logic because it’s way easier to scale services horizontally? With the MarkLogic approach, scaling the database is easy. But is there benefit to having complex queries in the database? I can think of two.

First, by keeping computation as close to the data as possible, we reduce the round-trips between the database and the application’s middle tier. The communication that does happen consists of the request and a final answer, instead of a larger amount of data that still needs processing.

Second, keeping complex queries in the database allows the middle tier to remain simpler. The middle tier makes simple calls to the database, without regard to how many nodes are in the database cluster.

Bottom line: if your database is hard to scale, maybe it’s worth looking into a different database.

Technical Community Manager

Start a discussion

Connect with the community

STACK OVERFLOW

EVENTS

GITHUB COMMUNITY

Most Recent

View All

Digital Acceleration Series: Powering MDM with MarkLogic

Our next event series covers key aspects of MDM including data integration, third-party data, data governance, and data security -- and how MarkLogic brings all of these elements together in one future-facing, agile MDM data hub.
Read Article

Of Data Warehouses, Data Marts, Data Lakes … and Data Hubs

New technology solutions arise in response to new business needs. Learn why a data hub platform makes the most sense for complex data.
Read Article

5 Key Findings from MarkLogic-Sponsored Financial Data Leaders Study

Financial institutions differ in their levels of maturity in managing and utilizing their enterprise data. To understand trends and winning strategies in getting the greatest value from this data, we recently co-sponsored a survey with the Financial Information Management WBR Insights research division.
Read Article
This website uses cookies.

By continuing to use this website you are giving consent to cookies being used in accordance with the MarkLogic Privacy Statement.