The new website for MarkLogic is www.progress.com/marklogic. Visit it.
BLOG ARTICLE

Mainframe Copybook to NoSQL

Back to blog
03.31.2014
2 minute read
Back to blog
03.31.2014
2 minute read
Person using a tablet

My colleague Ken Krupa recently found himself time-traveling between his early grunt years as a COBOL programmer and his current role as chief architect of an Enterprise NoSQL company. A meeting with a (huge) organization, heavily reliant on (massive) amounts of data generated from legacy mainframes had him playing with COBOL “copybooks” to bring them into MarkLogic.

COBOL has been around a few more years than Ken and was at the heart of many mission-critical systems across virtually every sector, from banking to manufacturing and healthcare to retail. Much of the structure lives in those aforementioned copybooks — sometimes embedded inside of the COBOL programs themselves. Getting this data into a NoSQL database would take the tarnish off this legacy data and spin it into a valuable commodity.

Legacy data structure from COBOL copybook.
Legacy data structure from COBOL copybook.

How hard was it to leapfrog from one architecture to another? “With MarkLogic and a set of open source Java libraries it was easy to ingest such mainframe data and also maintain the structure from the copybooks in a self-describing way via XML,” Ken told me. “More important however, having the data fully indexed in MarkLogic, regardless of the shape or structure of the data allows companies to glean new insights into such data like never before.”

Conversion of mainframe data files as defined by copybooks into a document-store like MarkLogic is a natural progression that was never well-suited in relational technologies.

Elegant XML is ingested into MarkLogic for easy analysis of legacy data.
Elegant XML is ingested into MarkLogic for easy analysis of legacy data.

“Copybooks are hierarchical by nature, as is XML,” Ken explained. “Also repeating items that might appear in an OCCURS clause of a copybook are handled easily with XML but are troublesome with relational. Additionally the poly-schema capabilities of NoSQL (e.g. different shapes of things with the same name) map well to the REDEFINES clause of copybooks. And then all of the typical stuff like not having to create tables or pre-define a landing place for the data just highlights the advantages further.”

An avid blogger, Ken chronicled his journey down memory lane — and  finding the right libraries to work his magic (and impress the prospect). In part 2 he shows the steps every step he took — including obfuscating actual data  by creating  a sample data set using custom copybook creators, the configuration of MarkLogic that anyone can download, mapping from COBOL to XSD. Questions? feel free to reach out to Ken via Twitter @kenkrupa.

 

Diane Burley

Responsible for overall content strategy and developing integrated content delivery systems for MarkLogic. She is a former online executive with Gannett with astute business sense, a metaphorical communication style and no fear of technology. Diane has delivered speeches to global audiences on using technologies to transform business. She believes that regardless of industry or audience, "unless the content is highly relevant -- and perceived to be valuable by the individual or organization -- it is worthless." 

Read more by this author

Share this article

Read More

Related Posts

Like what you just read, here are a few more articles for you to check out or you can visit our blog overview page to see more.

Tutorial

Poker Fun with XQuery

In this post, we dive into building a full five-card draw poker game with a configurable number of players. Written in XQuery 1.0, along with MarkLogic extensions to the language, this game provides examples of some great programming capabilities, including usage of maps, recursions, random numbers, and side effects. Hopefully, we will show those new to XQuery a look at the language that they may not get to see in other tutorials or examples.

All Blog Articles
Tutorial

Protecting passwords in ml-gradle projects

If you are getting involved in a project using ml-gradle, this tip should come in handy if you are not allowed to put passwords (especially the admin password!) in plain text. Without this restriction, you may have multiple passwords in your gradle.properties file if there are multiple MarkLogic users that you need to configure. Instead of storing these passwords in gradle.properties, you can retrieve them from a location where they’re encrypted using a Gradle credentials plugin.

All Blog Articles
Tutorial

Getting Started with Apache Nifi: Migrating from Relational to MarkLogic

Apache NiFi introduces a code-free approach of migrating content directly from a relational database system into MarkLogic. Here we walk you through getting started with migrating data from a relational database into MarkLogic

All Blog Articles

Sign up for a Demo

Don’t waste time stitching together components. MarkLogic combines the power of a multi-model database, search, and semantic AI technology in a single platform with mastering, metadata management, government-grade security and more.

Request a Demo