MarkLogic 7 Vision: World-Class Triple Store and World-Beating Information Store from MarkLogic.

The author is Demian Hess, from Avalon Consulting (a MarkLogic partner, who has built some very cool stuff on top of our database), and he describes what he sees as the difference between “Big Data” and “Cumbersome Data.” Check it out if you’ve got a minute!

Do you have Big Data or Cumbersome Data? from MarkLogic.

MarkLogic Looks to Push NoSQL Beyond Publishing from MarkLogic.

Credit card companies like Visa measure fraud in terms of identity theft and stolen cards: the cardholder is NOT the person making the purchase, and thus the transaction is fraudulent. In Medicaid, unscrupulous medical providers, sometimes in collaboration with compromised beneficiaries, are billing for services NOT rendered – or for patients who don’t exist. In both of these situations, the key is not in finding the computer network that abets the fraud – but finding the social network that masterminds it. And the key to that is in Big Data.

With the ubiquity of computers, it is hard for humans to avoid leaving electronic footprints. However the complexity of human behavior means tracking the right footprints to find intersections and correlations. This is no easy task for risk managers because integrating hundreds of datasets into relational models that could reveal fraudulent patterns has been expensive and time-consuming. According to The Wall Street Journal it took Visa years to expand its analytic engine from 40 aspects in 2005 to now handle 500. That expansion has significantly cut the risk of fraud. But it took years to get there. A pilot between MarkLogic and The Centers for Medicare & Medicaid Services (CMS) suggests there is a much better way and that’s good news for taxpayers.

Fraud in healthcare is massive. The FBI estimates that CMS was ripped off $80 billion in just 2012. CMS was able to recover a paltry $4.2 billion – and it cost $1.2 billion to do that. Fraud discovery has been nearly impossible because of all the handcuffs on the process — imposed by databases and regulations. Claims, Beneficiaries and Provider info all sit in different relational databases – each set up and maintained by individual states and territories. Medical billing is beyond complex. You can have 1000-line-item claims that get broken up into a hundreds of tables in a relational database, which then need to be joined.  Congress mandates that bills be paid within 30 days – and so it becomes a “pay and chase” routine. This traditionally high gain/low risk of being discovered has empowered criminals – but all that (finally) seems set to change.

Recently CMS teamed up with MarkLogic to find new more efficient means to detect fraud. MarkLogic ingested over 600M records comprised of multiple heterogeneous claims types and related data for a two year period. Because MarkLogic does not require data to conform to a single schema and uses a hierarchical tree structure to describe relationships the team was able to quickly load the various claims types and easily add data from external datasets, such as Dunn & Bradstreet Reports, National Provider Identifiers database, Facebook, multiple claims, extracts claims, DEA schedules, list of excluded individuals and entities (LEIE), diagnostic codes, indictment documents, SAS fraud algorithm reports and state policy documents. All of this was incorporated into rich hierarchical Provider profiles that presented a picture of many aspects of a providers behaviors and history. An interesting picture began to emerge.

As a result of the database and the application we built we found a number of aberrant behaviors. As my colleague Mike Doane likes to emphasize, MarkLogic developers had “no fraud experts on the team.” Bad “stuff just jumped out,” he said in a presentation on Innovative Fraud Detection at CMS given at MarkLogic World. “We found doctors with very high billings per year, a high percentage of high-cost procedures, too many people operating in one geospatial area, providers who showed up on exclusion lists (indictment documents).

“By using D&B data, they could see one owner of several different provider companies – all located at the same address, right down to the suite. Further, those provider companies had a high concentration of shared beneficiaries between those entities. Those shared beneficiaries may be innocent — or part of the network of thieves,” he explained.

With a successful pilot behind them, CMS is set to roll out a more comprehensive fraud discovery system. And Doane concluded, “while there’s lots of room for improvement to reduce and recover fraud – there is even more money CMS can save in reducing waste.” Stay tuned.

Fraud: Weapon of Asset Destruction from MarkLogic.

Let’s start by focusing on the main question: How can one guarantee cross-record ACID transactions in a horizontally-scalable, schema-agnostic database?

The short answer is an architectural pattern called Multi Version Concurrency Control or MVCC.

The basic notion behind MVCC is that records are never modified, but instead a new version is created every time a record changes. The system eventually deletes these old versions after a configureable period of time, but within that time window it’s simple to roll back a transaction. More over, it’s also straight forward to roll back the entire database to an earlier point in time – A.K.A. point-in-time recovery – a key requirement of enterprise databases.

Interestingly enough, the availability of Enterprise NoSQL – a schema-agnostic technology that satisfies these requirements – is now starting to blur the boundaries between the traditional Data Warehouse, Operational Data Store and DataMart, and converge them into a single store. The enabler for this is the notion of schema-on-read (vs. the traditional schema-on-write), which refers to the ability to enter data without requiring a pre-defined schema, while supporting multiple schemas when the data is read. This means that the categories mentioned above can be merged into a single platform that satisfies many data consumers without requiring intense modeling and transformation ahead of time.

In addition to schema-on-read, it is also the unification of data management and search that is key to handling data diversity. In fact it was the immense success of search engines that paved the way to this new data management paradigm. Search technologies have established the use of a rich set of indexes as a means for querying non-relational data. From there it was a small leap to apply this notion to a database, converging it with database indexing. But unlike traditional RDBMS, indexes in the NoSQL world do not have do be pre-defined, nor rebuilt as the data changes.

So we’re witnessing some related convergence trends – the convergence of structured and unstructured data, that of database and search technologies, and of traditional data management tiers into a single platform.

My next post will tie these concepts back to the related industry use-cases that benefit from them.

ACID, BASE and NoSQL from MarkLogic.

MarkLogic Points Up its Hadoop Big-data Future from MarkLogic.

“Technology continues to play a pivotal role in transforming how business and society functions. For the past 25 years The Computerworld Honors Program has had the privilege of celebrating innovative IT achievements,” said John Amato, vice president and publisher, Computerworld. “Computerworld is honored to recognize the outstanding accomplishments of the 2013 class of Laureates and to share their work. These projects demonstrate how IT can advance organizations’ ability to compete, innovate, communicate and prosper.”

“SAMHSA’s mission to reduce the impact of substance abuse and mental illness on America’s communities is an important effort. MarkLogic is proud to have assisted IQ Solutions in delivering an approach that expanded the organization’s ability to disseminate its valuable content,” said Kevin Shelly, vice president sales, Public Sector, MarkLogic. “MarkLogic’s ability to store, index, and search XML documents made it possible for IQ Solutions to successfully develop a structure for SAMHSA that could deliver any content to any platform.”

MarkLogic delivers a powerful and trusted enterprise-grade NoSQL database that enables organizations to turn all data into valuable and actionable information. A changing health information market and government mandates required SAMHSA to use shared platforms and focus on digital formats creating a need for a new, unique, and flexible content solution. IQ Solutions embraced a “create once, publish everywhere” philosophy for the SAMHSA solution, and leveraged MarkLogic’s technology to build a new XML content repository so that all of the organization’s content could be easily stored, searched, and shared.

“This award recognizes companies with leading edge technologies that are providing value to companies across industries and I was proud to nominate MarkLogic,” said Merritt Lutz, Senior Advisor, Morgan Stanley, and member of the Computerworld Honors Foundation Chairmen’s Committee.

The Computerworld Honors Program awards will be presented at the Gala Evening and Awards Ceremony on June 3, 2013 at the Andrew W. Mellon Auditorium in Washington, D.C.

About The Computerworld Honors Program

Founded by International Data Group (IDG) in 1988, The Computerworld Honors Program is governed by the not-for-profit Computerworld Information Technology Awards Foundation. Computerworld Honors is the longest running global program to honor individuals and organizations that use information technology to promote positive social, economic and educational change. Additional information about the program and a Global Archive of past Laureate case studies, as well as oral histories of Leadership Award recipients can be found at the Computerworld Honors website.

About Computerworld

Computerworld is the leading source of technology news and information for IT influencers, providing peer perspective, IT leadership and business results. Computerworld’s award-winning website (http://www.computerworld.com), bi-weekly publication, focused conference series, custom solutions and custom research forms the hub of the world’s largest (40+ edition) global IT media network and provides opportunities for IT solutions providers to engage this audience. Computerworld leads the industry with an online audience of over 3.5 million unique, monthly visitors (Omniture, August 2012) and was recognized as the Best Website by ASBPE and TABPI in 2012. Computerworld is published by IDG Enterprise, a subsidiary of International Data Group (IDG), the world’s leading media, events and research company. Company information is available at http://www.idgenterprise.com.

MarkLogic® Recognized as a 2013 Computerworld Honors Laureate from MarkLogic.

I decided to get myself to a place where the alphabet soup became a bit more palatable – but where to begin? Taking guidance from the words of Julie Andrews: “When you read you begin with A-B-C, when you sing you begin with Do-Re-Mi,” I believe that in the MarkLogic world it all begins with S-Q-L. This may seem counter-intuitive but it seems that unless you understand SQL then NOSQL will mean very little to you and Enterprise NOSQL even less.

According to Wikipedia SQL or Structured Query Language is a “special-purpose programming language designed for managing data held in relational database management systems (RDBMS).” Thanks to the likes of IBM and Oracle (formerly Relational Software Inc), SQL was developed and became commercially available in various systems by the end of the 1970s and early 1980s. It became ANSI (American National Standards Institute) standard in 1986 but in the tradition of all great standards it can differ slightly between databases and different vendors. SQL is used to define and manipulate data but was specifically designed as one of the first commercial languages for the relational database model. In very simplistic terms, this model uses tables of data in columns and rows which are related together using a primary key.

 (Enter NOSQL, Stage Right)

Let’s clear up the name first. NOSQL, originally meaning “No SQL”, is now generally acknowledged to be an acronym meaning “Not Only SQL”, since a number of databases of this type do allow SQL type language to be used.

My colleague David Cassel delivered a presentation on NOSQL and the various types of NOSQL databases and he explains that there are (roughly) four different types – Graph, Key Value Store, Column Store and Document Store. MarkLogic is one of the latter and this means that it uses identifiers called URIs and each of these has a document associated with it. A document is a unit of storage and is analogous to a row in a relational database. To see how they each stack up, check out the MarkLogic Founders Award winner Mike Bowers presentation on the pros and cons of each.

The key differentiator for NOSQL databases is that they do not use the relational model nor any fixed schema. They also, mostly, don’t use SQL to access data. This can lend a great deal of flexibility, performance and speed particularly when dealing with unstructured data – that is data that does not fit well into columns and rows without losing much of its meaning – for example an email, document or tweet.  In a world where Big Data has entered common parlance, NOSQL is coming into its own because we are constantly reminded that 80% of our data is unstructured and so a new generation of databases is needed to help deal with it.

So-fah-So-Good? Next time I will take a look at some acronyms that represent the features that make MarkLogic the only Enterprise NOSQL database.

*With thanks to Tony Hughes for inspiration.

‘Who are you and what do you do?’* from MarkLogic.

We’ll be doing this by participating in the CTE (Continuous Transformational Environment) Lab hosted within the QTS (Quality Technology Services) data center in Virginia. Leading hardware and software vendors are participating in the lab (it will feature a variety of COTS and GOTS products), and a launch event is being held on Friday May 3. The launch event will be attended by Virginia governor Robert McDonnell as well as key DoD leaders.

On an ongoing basis, the lab will be used to test out software and hardware interoperability and performance metrics in an environment that mimics deployment environments at scale. So, our government customers will be able to be confident that applications and infrastructure work as intended when deployed to production environments. Awesome!

The lab will be supporting several reference architectures, and MarkLogic will be included as a data layer providing a real-time, ACID catalog for Hadoop – basically, a Search, Alerting, and REST/JSON/XML DBMS “service dial tone” that customers and their contractors can easily tap into for delivery of data to their applications. A couple of MarkLogic team members will be on hand at the launch event to talk about these capabilities.

At the ceremony MarkLogic will discuss how establishing the CTE data layer will drive down the time and cost of implementing and testing Cloud applications. Additionally, we will demonstrate how this architecture can help provide scalable solutions for Watchroom personnel and analysts that have to pour through hundreds or thousands of record message traffic.

I am really excited that we’re going to be participating in this, both from a MarkLogic employee perspective as well as a U.S. taxpayer perspective – this lab has got such great potential to help the government save money (and time) in developing new, successful projects that will help protect and serve our country.

U.S. government gets a new test bed for secure cloud computing from MarkLogic.

There are countless examples of business value locked up in data that does not fit neatly into rows and columns. The most frequently cited is Social Media, with its ability to offer deep customer insight and sentiment analysis. And there are many others within the company’s firewall as well: Gleaning information from on-boarding documents for FATCA and AML compliance, getting a better handle on credit risk by analyzing ISDA agreements, lowering cost per trade by consolidating the processing of diverse asset classes with varied and complex structures, etc.

How can we effectively handle all this information, which is either hidden in free-form text, or scattered across incompatible schemas? Hierarchical structures such as XML and JSON certainly come to mind, as they can accommodate various degrees of structure, organized in a way that mirrors intuitive human perception. Indeed, many organizations have been using XML to handle these business challenges and have reaped some benefits, but found themselves constrained by the underlying RDBMS platforms that actually managed the data.

The problem with the typical approach to handling hierarchical information is that data is “shredded” into tables: a customer / derivative trade / legal document, with all its hierarchical attributes, is shoehorned into an ER model that satisfies referential integrity. Don’t get me wrong – I love relational modeling and I have spent years doing it, but 3rd Normal Form has its limitations when it comes to diverse data: just consider the typical first step when analyzing normalized data: de-normalize it!

There is an alternative to shredding though, in the form of NoSQL – a wide set of technologies that transcend the boundaries of relational schemas. The name is somewhat unfortunate since SQL is actually one of the best features associated with an RDBMS (some call it the most successful Domain Specific Language). The problem with RDBMSs is not SQL but the prerequisite of a schema definition for data ingestion and analysis, which hinders business agility. We’ve all seen cases where the business needs have been delayed while data models, transformations and analytical schemas were being developed. NoSQL databases free us from the rains of the schema to enable real business agility.

However, one factor has prevented a wide adoption of NoSQL technologies within the enterprise: the BASE architectural principle underlying most of them. It stands for Basically Available, Soft state, Eventual Consistency – a play on ACID transactions (Atomicity, Consistency, Isolation, Durability), which are associated with relational databases. BASE has several advantages when it comes to non-transactional systems, as it relaxes consistency to allow the system to process requests even in an inconsistent state. Social media sites are a perfect example – No one would mind if their Facebook status or latest tweet were inconsistent within their social network for a short period of time; it’s much more important to get an immediate response than to have a consistent state of users’ information.

Financial and other enterprise systems are a different matter though. Imagine for instance, a merger corporate action, occurring at the same time a firm is trading the affected instrument: The post-trade processing systems would certainly have to be consistent with the Reference Data system, or costly exceptions would ensue.

So how do we avoid schema woes without giving up ACID transactions, as well other enterprise qualities such as fine-grained entitlements, point-in-time recovery, and high availability, all of which we’ve come to expect for mission-critical system?

The answer lies within a different category of technology called Enterprise NoSQL, which has been designed and built with transactions and enterprise features from the ground up, just like relational databases. But unlike RDMBS, an Enterprise NoSQL database models the data as hierarchical trees rather than rows and columns. These trees are aggressively indexed in-memory as soon as the data is ingested, and then used for both element retrieval and full text search, unifying two concepts that have traditionally been separate – the database and the search engine.

An Enterprise NoSQL database also offers full SQL access, thus combining the benefits of both worlds – the business agility associated with NoSQL and search, and the data integrity and sophisticated querying associated with a traditional RDBMS.

In the next installment of this blog I will explore the mechanisms by which this is achieved.

Enterprise Big Data: It’s Not About Size from MarkLogic.