Gartner Cloud DBMS Report Names MarkLogic a Visionary

When to be lazy, and when to be eager, when querying the database

Generally, optimizations in the evaluator try to be lazy whenever they can when evaluating functions. This allows the application to process data only when needed and limits memory use to only what is needed. But sometimes, you need to evaluate eagerly to improve performance. Here we discuss one use case when it is best to specify eager evaluation.

Let’s say you need to write a script that collects all the values from a range index and groups them into “sets” of 1000. We may end up with something like this:

xquery version "1.0-ml";
declare namespace sample = "http://marklogic.com/sample";
let $groupsize := 1000
let $values := cts:element-values(xs:QName("sample:value"))
let $count := fn:count($values)

let $groups :=
    for $i in (0 to (xs:int($count div $groupsize) + 1))
    let $group := $values[(($i * $groupsize) + 1) to (($i + 1) * $groupsize)]
    return fn:string-join($group, "|")

return (fn:count($groups), $groups)

This query performs fine on a small set of values (5000), but when we increase the number of values pulled from the range index, we see that this call

let $group := $values[(($i * $groupsize) + 1) to (($i + 1) * $groupsize)]

quickly becomes the long pole in the tent. In fact, for a sample size of 50,000 values (50 groups), 91% of the execution time is taken by this one call, 2.3 seconds for just 50 calls. Increasing the sample size to values above 1,000,000 and it’s clear that this query will no longer run in a reasonable amount of time. So what’s going on here?

As it turns out, our cts:element-values() call is returning a stream, which is loaded lazily as needed. Therefore, fifty sequence accesses are actually fifty stream accesses, each time streaming the results from the first item (grabbing items at the end of the stream takes longer).

Because we are iterating through the groups, this is a situation where we want eager evaluation, ensuring that you get all of your data back in-memory all at once before you begin to iterate through the data. The function cts:element-values() has an “eager” options flag. Another easy “trick” that will reliably ensure you’re working with the full in-memory sequence is to simply drop into a “sub-flwor” statement to generate a sequence from the return value:

let $values := for $i in cts:element-values(xs:QName("sample:value")) return $i

Now, $values holds the entire resulting dataset and now accessing subsequences within it is much faster, particularly values at the end of the sequence.

 

Additional Resources

Start a discussion

Connect with the community

STACK OVERFLOW

EVENTS

GITHUB COMMUNITY

Most Recent

View All

Multi-Model: The Next Step In Database Technology?

Does your database just store multiple data types, or can it also serve them back to the business with agility and flexibility? Get some tips on things to consider when evaluating multi-model database technologies.
Read Article

What Makes Complex Data Different

How do you know when you have complex data, and why is it important? Industry veteran Chuck Hollis explains.
Read Article

Standardizing Internal Data Models on FHIR

Learn about MarkLogic's work on a FHIR-based standardized data model to support persisted payer data for our Medicaid Accelerators.
Read Article
This website uses cookies.

By continuing to use this website you are giving consent to cookies being used in accordance with the MarkLogic Privacy Statement.