Parsing broken XML with lxml.etree.iterparse

Question

I'm trying to parse a huge xml file with lxml in a memory efficient manner (ie streaming lazily from disk instead of loading the whole file in memory). Unfortunately, the file contains some bad ascii characters that break the default parser. The parser works if I set recover=True, but the iterparse method doesn't take the recover parameter or a custom parser object. Does anyone know how to use iterparse to parse broken xml?

#this works, but loads the whole file into memory
parser = lxml.etree.XMLParser(recover=True) #recovers from bad characters.
tree = lxml.etree.parse(filename, parser)

#how do I do the equivalent with iterparse?  (using iterparse so the file can be streamed lazily from disk)
context = lxml.etree.iterparse(filename, tag='RECORD')
#record contains 6 elements that I need to extract the text from

Thanks for your help!

EDIT -- Here is an example of the types of encoding errors I'm running into:

In [17]: data
Out[17]: '\t<articletext>&lt;p&gt;The cafeteria rang with excited voices.  Our barbershop quartet, The Bell \r Tones was asked to perform at the local Home for the Blind in the next town.  We, of course, were glad to entertain such a worthy group and immediately agreed .  One wag joked, "Which uniform should we wear?"  followed with, "Oh, that\'s right, they\'ll never notice."  The others didn\'t respond to this, in fact, one said that we should wear the nicest outfit we had.&lt;/p&gt;&lt;p&gt;A small stage was set up for us and a pretty decent P.A. system was donated for the occasion.  The audience was made up of blind persons of every age, from the thirties to the nineties.  Some sported sighted companions or nurses who stood or sat by their side, sharing the moment equally.  I observed several German shepherds lying at their feet, adoration showing in their eyes as they wondered what was going on.  After a short introduction in which we identified ourselves, stating our voice part and a little about our livelihood, we began our program.  Some songs were completely familiar and others, called "Oh, yeah" songs, only the chorus came to mind.  We didn\'t mind at all that some sang along \x1e they enjoyed it so much.&lt;/p&gt;&lt;p&gt;In fact, a popular part of our program is when the audience gets to sing some of the old favorites.  The harmony parts were quite evident as they tried their voices to the different parts.  I think there was more group singing in the old days than there is now, but to blind people, sound and music is more important.   We received a big hand at the finale and were made to promise to return the following year.  Everyone was treated to coffee and cake, our quartet going around to the different circles of friends to sing a favorite song up close and personal.  As we approached a new group, one blind lady amazed me by turning to me saying, "You\'re the baritone, aren\'t you?"  Previously no one had ever been able to tell which singer sang which part but this lady was listening with her whole heart.&lt;/p&gt;&lt;p&gt;Retired portrait photographer.  Main hobby - quartet singing.&lt;/p&gt;</articletext>\n'

In [18]: lxml.etree.from
lxml.etree.fromstring      lxml.etree.fromstringlist  

In [18]: lxml.etree.fromstring(data)
---------------------------------------------------------------------------
XMLSyntaxError                            Traceback (most recent call last)

/mnt/articles/<ipython console> in <module>()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree.fromstring (src/lxml/lxml.etree.c:48270)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:71812)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._parseDoc (src/lxml/lxml.etree.c:70673)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:67442)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:63824)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64745)()

/usr/lib/python2.5/site-packages/lxml-2.2.4-py2.5-linux-i686.egg/lxml/etree.so in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)()

XMLSyntaxError: PCDATA invalid Char value 30, line 1, column 1190

In [19]: chardet.detect(data)
Out[19]: {'confidence': 1.0, 'encoding': 'ascii'}

As you can see, chardet thinks it is an ascii file, but there is a "\x1e" right in the middle of this example which is making lxml raise an exception.

Solution 1

I solved the problem by creating a class with a File like object interface. The class' read() method reads a line from the file and replaces any "bad characters" before returning the line to iterparse.

#psudo code

class myFile(object):
    def __init__(self, filename):
        self.f = open(filename)

    def read(self, size=None):
        return self.f.next().replace('\x1e', '').replace('some other bad character...' ,'')


#iterparse
context = lxml.etree.iterparse(myFile('bigfile.xml', tag='RECORD')

I had to edit the myFile class a few times adding some more replace() calls for a few other characters that were making lxml choke. I think lxml's SAX parsing would have worked as well (seems to support the recover option), but this solution worked like a charm!

Solution 2

Edit:

This is an older answer and I would have done it differently today. And I'm not just referring to the dumb snark ... since then BeutifulSoup4 is available and it's really quite nice. I recommend that to anyone who stumbles over here.

---

The currently accepted answer is, well, not what one should do. The question itself also has a bad assumption:

parser = lxml.etree.XMLParser(recover=True) #recovers from bad characters.

Actually recover=True is for recovering from misformed XML. There is however an "encoding" option which would have fixed your issue.

parser = lxml.etree.XMLParser(encoding='utf-8' #Your encoding issue.
                              recover=True, #I assume you probably still want to recover from bad xml, it's quite nice. If not, remove.
                              )

That's it, that's the solution.

---

BTW -- For anyone struggling with parsing XML in python, especially from third party sources. I know, I know, the documentation is bad and there are a lot of SO red herrings; a lot of bad advice.

• lxml.etree.fromstring()? - That's for perfectly formed XML, silly
• BeautifulStoneSoup? - Slow, and has a way-stupid policy for self closing tags
• lxml.etree.HTMLParser()? - (because the xml is broken) Here's a secret - HTMLParser() is... a Parser with recover=True
• lxml.html.soupparser? - The encoding detection is supposed to be better, but it has the same failings of BeautifulSoup for self closing tags. Perhaps you can combine XMLParser with BeautifulSoup's UnicodeDammit
• UnicodeDammit and other cockamamie stuff to fix encodings? - Well, UnicodeDammit is kind of cute, I like the name and it's useful for stuff beyond xml, but things are usually fixed if you do the right thing with XMLParser()

You could be trying all sorts of stuff from what's available online. lxml documentation could be better. The code above is what you need for 90% of your XML parsing cases. Here I'll restate it:

magical_parser = XMLParser(encoding='utf-8', recover=True)
tree = etree.parse(StringIO(your_xml_string), magical_parser) #or pass in an open file object

You're welcome. My headaches == your sanity. Plus it has other features you might need for, you know, XML.

Solution 3

Edit your question, stating what happens (exact error message and traceback (copy/paste, don't type from memory)) to make you think that "bad unicode" is the problem.

Get chardet and feed it your MySQL dump. Tell us what it says.

Show us the first 200 to 300 bytes of your dump, using e.g. print repr(dump[:300])

Update You wrote """As you can see, chardet thinks it is an ascii file, but there is a "\x1e" right in the middle of this example which is making lxml raise an exception."""

I see no "bad unicode" here.

chardet is correct. What makes you think that "\x1e" is not ASCII? It is an ASCII character, a C0 control character named "RECORD SEPARATOR".

The error message says that you have an invalid character. That is also correct. The only control characters that are valid in XML are "\t", "\r" and "\n". MySQL should be grumbling about that and/or offering you a way of escaping it e.g. _x001e_ (yuk!)

Given the context, it looks like that character could be deleted with no loss. You may wish to fix your database or you may wish to remove suchlike characters from your dump (after checking that they are all vanishable) or you may wish to choose a less picky and less volumnious output format than XML.

Update 2 You presumably want to user iterparse() not because it's your end goal but because you want to save memory. If you used a format like CSV you wouldn't have a memory problem.

Update 3 In response to a comment by @Purrell:

try it yourself, dude. pastie.org/3280965

Here's the contents of that pastie; it deserves preservation:

from lxml.etree import etree

data = '\t<articletext>&lt;p&gt;The cafeteria rang with excited voices.  Our barbershop quartet, The Bell \r Tones was asked to perform at the local Home for the Blind in the next town.  We, of course, were glad to entertain such a worthy group and immediately agreed .  One wag joked, "Which uniform should we wear?"  followed with, "Oh, that\'s right, they\'ll never notice."  The others didn\'t respond to this, in fact, one said that we should wear the nicest outfit we had.&lt;/p&gt;&lt;p&gt;A small stage was set up for us and a pretty decent P.A. system was donated for the occasion.  The audience was made up of blind persons of every age, from the thirties to the nineties.  Some sported sighted companions or nurses who stood or sat by their side, sharing the moment equally.  I observed several German shepherds lying at their feet, adoration showing in their eyes as they wondered what was going on.  After a short introduction in which we identified ourselves, stating our voice part and a little about our livelihood, we began our program.  Some songs were completely familiar and others, called "Oh, yeah" songs, only the chorus came to mind.  We didn\'t mind at all that some sang along \x1e they enjoyed it so much.&lt;/p&gt;&lt;p&gt;In fact, a popular part of our program is when the audience gets to sing some of the old favorites.  The harmony parts were quite evident as they tried their voices to the different parts.  I think there was more group singing in the old days than there is now, but to blind people, sound and music is more important.   We received a big hand at the finale and were made to promise to return the following year.  Everyone was treated to coffee and cake, our quartet going around to the different circles of friends to sing a favorite song up close and personal.  As we approached a new group, one blind lady amazed me by turning to me saying, "You\'re the baritone, aren\'t you?"  Previously no one had ever been able to tell which singer sang which part but this lady was listening with her whole heart.&lt;/p&gt;&lt;p&gt;Retired portrait photographer.  Main hobby - quartet singing.&lt;/p&gt;</articletext>\n'

magical_parser = etree.XMLParser(encoding='utf-8', recover=True)
tree = etree.parse(StringIO(data), magical_parser)

To get it to run, one import needs to be fixed, and another supplied. The data is monstrous. There is no output to show the result. Here's a replacement with the data cut down to the bare essentials. The 5 pieces of ASCII text (excluding < and >) that are all valid XML characters are replaced by t1, ..., t5. The offending \x1e is flanked by t2 and t3.

[output wraps at column 80]
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>> from cStringIO import StringIO
>>> data = '<article>&lt;p&gt;t1&lt;/p&gt;&lt;p&gt;t2\x1et3&lt;/p&gt;&lt;p&gt;t4
&lt;/p&gt;&lt;p&gt;t5&lt;/p&gt;</article>'
>>> magical_parser = etree.XMLParser(encoding='utf-8', recover=True)
>>> tree = etree.parse(StringIO(data), magical_parser)
>>> print(repr(tree.getroot().text))
'<p>t1</p><p>t2t3/ppt4/ppt5/p'

Not what I'd call "recovery"; after the bad character, the < and > characters disappear.

The pastie was in response to my question "What gives you the idea that encoding='utf-8' will solve his problem?". This was triggered by the statement 'There is however an "encoding" option which would have fixed your issue.' But encoding=ascii produces the same output. So does omitting the encoding arg. It's NOT an encoding problem. Case closed.

Comments

[erikcw]

I've added that info to my question... Thanks for your help!

[erikcw]

Thanks for the info! I'd still like to try to figure out a way to make lxml.etree.iterparse fix the broken XML just like lxml.etree.parse(filename, lxml.etree.XMLParser(recover=True)) is able to.

[erikcw]

@John Machin: RE: "Update 2" -- Yes, I'm using iterparse() to save memory. It's true that in this case I could dump the MySQL db into CSV. That is a very practical solution. However, I'd still like to know how to handle this issue with lxml since processing large XML documents is something I'll likely have to do in the future. It's a very generic issue don't you think?

[John Machin]

@ericw: How to handle this issue with lxml: (1) you change the lxml code to do what you want (2) you pay someone else (e.g. the lxml author) to do what you want (3) you sit on the beach until someone else changes lxml to do what you want -- all conditional on whether it's possible/sensible for lxml to be so changed. The principled solution for processing large XML documents is to make them valid XML before you attempt to parse them; that way the recovery is under your control.

[John Machin]

-1 What gives you the idea that encoding='utf-8' will solve his problem? He reports that his data contains "some sang along \x1e they enjoyed" ... \x1e is a valid ASCII character and thus a valid UTF-8 character. His problem is that \x1e is not a valid character in XML.

[8bitjunkie]

lxml.etree.fromstring() appears to happily pass broken XML in v3.3.0 and begins to become strict from v3.5.0 upwards.

[minion]

What about lxml.html.fromstring()? You didn't mention it.

[kororo]

I am from 2018. Great answer. Dammit XML.

[Kent Wong]

What about std lib like Element Tree?