We’ve joined forces with Smartlogic to reveal smarter decisions—together.

Mainframe Copybook to NoSQL

My colleague Ken Krupa recently found himself time-traveling between his early grunt years as a COBOL programmer and his current role as chief architect of an Enterprise NoSQL company. A meeting with a (huge) organization, heavily reliant on (massive) amounts of data generated from legacy mainframes had him playing with COBOL “copybooks” to bring them into MarkLogic.

COBOL has been around a few more years than Ken and was at the heart of many mission-critical systems across virtually every sector, from banking to manufacturing and healthcare to retail. Much of the structure lives in those aforementioned copybooks — sometimes embedded inside of the COBOL programs themselves. Getting this data into a NoSQL database would take the tarnish off this legacy data and spin it into a valuable commodity.

Legacy data structure from COBOL copybook.
Legacy data structure from COBOL copybook.

How hard was it to leapfrog from one architecture to another? “With MarkLogic and a set of open source Java libraries it was easy to ingest such mainframe data and also maintain the structure from the copybooks in a self-describing way via XML,” Ken told me. “More important however, having the data fully indexed in MarkLogic, regardless of the shape or structure of the data allows companies to glean new insights into such data like never before.”

Conversion of mainframe data files as defined by copybooks into a document-store like MarkLogic is a natural progression that was never well-suited in relational technologies.

Elegant XML is ingested into MarkLogic for easy analysis of legacy data.
Elegant XML is ingested into MarkLogic for easy analysis of legacy data.

“Copybooks are hierarchical by nature, as is XML,” Ken explained. “Also repeating items that might appear in an OCCURS clause of a copybook are handled easily with XML but are troublesome with relational. Additionally the poly-schema capabilities of NoSQL (e.g. different shapes of things with the same name) map well to the REDEFINES clause of copybooks. And then all of the typical stuff like not having to create tables or pre-define a landing place for the data just highlights the advantages further.”

An avid blogger, Ken chronicled his journey down memory lane — and  finding the right libraries to work his magic (and impress the prospect). In part 2 he shows the steps every step he took — including obfuscating actual data  by creating  a sample data set using custom copybook creators, the configuration of MarkLogic that anyone can download, mapping from COBOL to XSD. Questions? feel free to reach out to Ken via Twitter @kenkrupa.

 

Chief Content Strategist

Responsible for overall content strategy and developing integrated content delivery systems for MarkLogic. She is a former online executive with Gannett with astute business sense, a metaphorical communication style and no fear of technology. Diane has delivered speeches to global audiences on using technologies to transform business. She believes that regardless of industry or audience, "unless the content is highly relevant -- and perceived to be valuable by the individual or organization -- it is worthless." 

Start a discussion

Connect with the community

STACK OVERFLOW

EVENTS

GITHUB COMMUNITY

Most Recent

View All

Unifying Data, Metadata, and Meaning

We're all drowning in data. Keeping up with our data - and our understanding of it - requires using tools in new ways to unify data, metadata, and meaning.
Read Article

How to Achieve Data Agility

Successfully responding to changes in the business landscape requires data agility. Learn what visionary organizations have done, and how you can start your journey.
Read Article

Scaling Memory in MarkLogic Server

This not-too-technical article covers a number of questions about MarkLogic Server and its use of memory. Learn more about how MarkLogic uses memory, why you might need more memory, when you need more memory, and how you can add more memory.
Read Article
This website uses cookies.

By continuing to use this website you are giving consent to cookies being used in accordance with the MarkLogic Privacy Statement.