My colleague Ken Krupa recently found himself time-traveling between his early grunt years as a COBOL programmer and his current role as chief architect of an Enterprise NoSQL company. A meeting with a (huge) organization, heavily reliant on (massive) amounts of data generated from legacy mainframes had him playing with COBOL “copybooks” to bring them into MarkLogic.
COBOL has been around a few more years than Ken and was at the heart of many mission-critical systems across virtually every sector, from banking to manufacturing and healthcare to retail. Much of the structure lives in those aforementioned copybooks — sometimes embedded inside of the COBOL programs themselves. Getting this data into a NoSQL database would take the tarnish off this legacy data and spin it into a valuable commodity.
How hard was it to leapfrog from one architecture to another? “With MarkLogic and a set of open source Java libraries it was easy to ingest such mainframe data and also maintain the structure from the copybooks in a self-describing way via XML,” Ken told me. “More important however, having the data fully indexed in MarkLogic, regardless of the shape or structure of the data allows companies to glean new insights into such data like never before.”
Conversion of mainframe data files as defined by copybooks into a document-store like MarkLogic is a natural progression that was never well-suited in relational technologies.
“Copybooks are hierarchical by nature, as is XML,” Ken explained. “Also repeating items that might appear in an OCCURS clause of a copybook are handled easily with XML but are troublesome with relational. Additionally the poly-schema capabilities of NoSQL (e.g. different shapes of things with the same name) map well to the REDEFINES clause of copybooks. And then all of the typical stuff like not having to create tables or pre-define a landing place for the data just highlights the advantages further.”
An avid blogger, Ken chronicled his journey down memory lane — and finding the right libraries to work his magic (and impress the prospect). In part 2 he shows
the steps every step he took — including obfuscating actual data by creating a sample data set using custom copybook creators, the configuration of MarkLogic that anyone can download, mapping from COBOL to XSD. Questions? feel free to reach out to Ken via Twitter @kenkrupa.
Like what you just read, here are a few more articles for you to check out or you can visit our blog overview page to see more.
In this post, we dive into building a full five-card draw poker game with a configurable number of players. Written in XQuery 1.0, along with MarkLogic extensions to the language, this game provides examples of some great programming capabilities, including usage of maps, recursions, random numbers, and side effects. Hopefully, we will show those new to XQuery a look at the language that they may not get to see in other tutorials or examples.
If you are getting involved in a project using ml-gradle, this tip should come in handy if you are not allowed to put passwords (especially the admin password!) in plain text. Without this restriction, you may have multiple passwords in your gradle.properties file if there are multiple MarkLogic users that you need to configure. Instead of storing these passwords in gradle.properties, you can retrieve them from a location where they’re encrypted using a Gradle credentials plugin.
Apache NiFi introduces a code-free approach of migrating content directly from a relational database system into MarkLogic. Here we walk you through getting started with migrating data from a relational database into MarkLogic
Don’t waste time stitching together components. MarkLogic combines the power of a multi-model database, search, and semantic AI technology in a single platform with mastering, metadata management, government-grade security and more.Request a Demo