We quite often see customers run into the exception invalid char between encapsulated token and delimiter when they are ingesting delimited text into MarkLogic Server using the MarkLogic Content Pump (MLCP). The error sounds technical and hard to understand — but what exactly is wrong with the data? Luckily, it’s not too hard to figure out how it happens and and how to solve it.
Invalid char between encapsulated token and delimiter means that you have invalid characters between an encapsulator and a delimiter. Hold on — what is an encapsulator? To put simply, it is the character used to wrap the CSV field or column that may contain special characters, such as line breaks. In most cases, people use double-quotes as the encapsulator.
For more details about encapsulators, please refer to the Internet Engineering Task Force (IETF) standard for CSVs.
Let’s explain with a delimited text example. Consider the following scenarios:
Here, the delimiter is “|” and the encapsulator is a double-quote. In rows 2, 3 and 4, there are some characters (bolded) between a delimiter and an encapsulator — here is where things went wrong. According to the IETF standard, those columns are actually not in a valid format for delimited text, which results in errors:
“Each field may or may not be enclosed in double quotes. If fields are not enclosed with double quotes, then double quotes may not appear inside the fields.”
Which brings up two points:
With that being said, rows 2, 3, 4 should be rejected by CSVParser as invalid CSV records. However, the CSVParser that MLCP currently uses can actually handle cases 2 and 3, parsing them without any issue; however, it is not able to deal with case 4, and in turn, will throw an exception with the message invalid char between encapsulated token and delimiter.
The best way to get around this exception is to avoid having malformed CSV data in the first place. If that is not possible, you can escape the double quotes in the field if you really want them to be part of the string. But remember, you must escape double quotes using another double quote in CSV! You CSV data will look something like this:
View all posts from Matt Sun on the Progress blog. Connect with us about all things application development and deployment, data integration and digital business.
Let our experts teach you how to use Sitefinity's best-in-class features to deliver compelling digital experiences.
Learn MoreSubscribe to get all the news, info and tutorials you need to build better business apps and sites
Progress collects the Personal Information set out in our Privacy Policy and the Supplemental Privacy notice for residents of California and other US States and uses it for the purposes stated in that policy.
You can also ask us not to share your Personal Information to third parties here: Do Not Sell or Share My Info
We see that you have already chosen to receive marketing materials from us. If you wish to change this at any time you may do so by clicking here.
Thank you for your continued interest in Progress. Based on either your previous activity on our websites or our ongoing relationship, we will keep you updated on our products, solutions, services, company news and events. If you decide that you want to be removed from our mailing lists at any time, you can change your contact preferences by clicking here.