How do I get the whole content between two xml tags in Python?

Question

I try to get the whole content between an opening xml tag and it's closing counterpart.

Getting the content in straight cases like title below is easy, but how can I get the whole content between the tags if mixed-content is used and I want to preserve the inner tags?

<?xml version="1.0" encoding="UTF-8"?>
<review>
  <title>Some testing stuff</title>
  <text sometimes="attribute">Some text with <extradata>data</extradata> in it.
  It spans <sometag>multiple lines: <tag>one</tag>, <tag>two</tag> 
  or more</sometag>.</text>
</review>

What I want is the content between the two text tags, including any tags: Some text with <extradata>data</extradata> in it. It spans <sometag>multiple lines: <tag>one</tag>, <tag>two</tag> or more</sometag>.

For now I use regular expressions but it get's kinda messy and I don't like this approach. I lean towards a XML parser based solution. I looked over minidom, etree, lxml and BeautifulSoup but couldn't find a solution for this case (whole content, including inner tags).

Solution 1

from lxml import etree
t = etree.XML(
"""<?xml version="1.0" encoding="UTF-8"?>
<review>
  <title>Some testing stuff</title>
  <text>Some text with <extradata>data</extradata> in it.</text>
</review>"""
)
(t.text + ''.join(map(etree.tostring, t))).strip()

The trick here is that t is iterable, and when iterated, yields all child nodes. Because etree avoids text nodes, you also need to recover the text before the first child tag, with t.text.

In [50]: (t.text + ''.join(map(etree.tostring, t))).strip()
Out[50]: '<title>Some testing stuff</title>\n  <text>Some text with <extradata>data</extradata> in it.</text>'

Or:

In [6]: e = t.xpath('//text')[0]

In [7]: (e.text + ''.join(map(etree.tostring, e))).strip()
Out[7]: 'Some text with <extradata>data</extradata> in it.'

Solution 2

Here's something that works for me and your sample:

from lxml import etree
doc = etree.XML(
"""<?xml version="1.0" encoding="UTF-8"?>
<review>
  <title>Some testing stuff</title>
  <text>Some text with <extradata>data</extradata> in it.</text>
</review>"""
)

def flatten(seq):
  r = []
  for item in seq:
    if isinstance(item,(str,unicode)):
      r.append(unicode(item))
    elif isinstance(item,(etree._Element,)):
      r.append(etree.tostring(item,with_tail=False))
  return u"".join(r)

print flatten(doc.xpath('/review/text/node()'))

Yields:

Some text with <extradata>data</extradata> in it.

The xpath selects all child nodes of the <text> element and either renders them to unicode directly if they are a string/unicode subclass (<class 'lxml.etree._ElementStringResult'>) or calls etree.tostring on it if it's an Element, with_tail=False avoids duplication of the tail.

You may need to handle other node types if they are present.

Solution 3

That is considerably easy with lxml*, using the parse() and tostring() functions:

from  lxml.etree import parse, tostring

First you parse the doc and get your element (I am using XPath, but you can use whatever you want):

doc = parse('test.xml')
element = doc.xpath('//text')[0]

The tostring() function returns a text representation of your element:

>>> tostring(element)
'<text>Some <text>text</text> with <extradata>data</extradata> in it.</text>\n'

However, you do not want the external elements, so we can remove them with a simple str.replace() call:

>>> tostring(element).replace('<%s>'%element.tag, '', 1)
'Some <text>text</text> with <extradata>data</extradata> in it.</text>\n'

Note that str.replace() received 1 as the third parameter, so it will remove only the first occurrence of the opening tag. One can do it with the closing tag, too. Now, instead of 1, we pass -1 to replace:

>>> tostring(element).replace('</%s>'%element.tag, '', -1)
'<text>Some <text>text with <extradata>data</extradata> in it.\n'

The solution, of course, is to do everything at once:

>>> tostring(element).replace('<%s>'%element.tag, '', 1).replace('</%s>'%element.tag, '', -1)
'Some <text>text with <extradata>data</extradata> in it.\n'

EDIT: @Charles made a good point: this code is fragile since the tag can have attributes. A possible yet still limited solution is to split the string at the first >:

>>> tostring(element).split('>', 1)
['<text',
 'Some <text>text</text> with <extradata>data</extradata> in it.</text>\n']

get the second resulting string:

>>> tostring(element).split('>', 1)[1]
'Some <text>text</text> with <extradata>data</extradata> in it.</text>\n'

then rsplitting it:

>>> tostring(element).split('>', 1)[1].rsplit('</', 1)
['Some <text>text</text> with <extradata>data</extradata> in it.', 'text>\n']

and finally getting the first result:

>>> tostring(element).split('>', 1)[1].rsplit('</', 1)[0]
'Some <text>text</text> with <extradata>data</extradata> in it.'

Nonetheless, this code is still fragile, since > is a perfectly valid char in XML, even inside attributes.

Anyway, I have to acknowledge that MattH solution is the real, general solution.

* Actually this solution works with ElementTree, too, which is great if you do not want to depend upon lxml. The only difference is that you will have no way of using XPath.

Solution 4

I like @Marcin's solution above, however I found that when using his 2nd option (converting a sub-node, not the root of the tree) it does not handle entities.

His code from above (modified to add an entity):

from lxml import etree
t = etree.XML("""<?xml version="1.0" encoding="UTF-8"?>
<review>
  <title>Some testing stuff</title>
    <text>this &amp; that.</text>
</review>""")
e = t.xpath('//text')[0]
print (e.text + ''.join(map(etree.tostring, e))).strip()

returns:

this & that.

with a bare/unescaped '&' character instead of a proper entity ('&').

My solution was to use to call etree.tostring at the node level (instead of on all children), then strip off the starting and ending tag using a regular expression:

import re
from lxml import etree
t = etree.XML("""<?xml version="1.0" encoding="UTF-8"?>
<review>
  <title>Some testing stuff</title>
    <text>this &amp; that.</text>
</review>""")

e = t.xpath('//text')[0]
xml = etree.tostring(e)
inner = re.match('<[^>]*?>(.*)</[^>]*>\s*$', xml, flags=re.DOTALL).group(1)
print inner

produces:

this & that.

I used re.DOTALL to ensure this works for XML containing newlines.

View more solutions

Comments

[Brutus]

I can get the same - I think - with x.find('text').get_text(). But this approach excludes the inner tags and I need them.

[brandizzi]

This does not solve the OP problem in any way, actually. It is required to maintain the inner tags.

[Charles Duffy]

The text replacement is adding quite a lot of fragility here. If your input file happens to have attributes on it? A namespace prefix?

[dav1d]

It does maintain the inner tags, just not more than one level, see my edit, itertext get's everything

[Charles Duffy]

This could be written more compactly. Take this one-liner: ''.join(el if isinstance(el, str) else lxml.etree.tostring(el, with_tail=False) for el in doc.xpath('/review/text/node()'))

[Brutus]

I have the feeling that I won't gain much over pure regular expressions with this approach. Since the opening tag has at least one attribute it get's flaky too.

[Marcin]

Iterate over all children, not just the text.

[Marcin]

You could probably just use tostring indiscriminately.

[MattH]

@Marcin: when I tried it, tostring complained that it couldn't serialize an _ElementStringResult

[Brutus]

replace('</%s>'%element.tag, '', -1) should work, but I can't use .replace('<%s>'%element.tag, '', 1) because there are one or more attributes, so I have to use regex again (or something around content[content.index('>'):]), etc.

[brandizzi]

The OP wants to get the content of a specific element. Your solution does not work in this case, at least not directly. I I get an element with e = t.xpath('//text')[0] and tried it (''.join(map(etree.tostring, e))) but the result was '<extradata>data</extradata> in it.'.

[Brutus]

Need to test on some more cases, but your last example works fine for me (so far). When using find instead of xpath it works with the standard etree too.