How do I use xml namespaces with find/findall in lxml?

Question

I'm trying to parse content in an OpenOffice ODS spreadsheet. The ods format is essentially just a zipfile with a number of documents. The content of the spreadsheet is stored in 'content.xml'.

import zipfile
from lxml import etree

zf = zipfile.ZipFile('spreadsheet.ods')
root = etree.parse(zf.open('content.xml'))

The content of the spreadsheet is in a cell:

table = root.find('.//{urn:oasis:names:tc:opendocument:xmlns:table:1.0}table')

We can also go straight for the rows:

rows = root.findall('.//{urn:oasis:names:tc:opendocument:xmlns:table:1.0}table-row')

The individual elements know about the namespaces:

>>> table.nsmap['table']
'urn:oasis:names:tc:opendocument:xmlns:table:1.0'

How do I use the namespaces directly in find/findall?

The obvious solution does not work.

Trying to get the rows from the table:

>>> root.findall('.//table:table')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml.etree.pyx", line 1792, in lxml.etree._ElementTree.findall (src/lxml/lxml.etree.c:41770)
  File "lxml.etree.pyx", line 1297, in lxml.etree._Element.findall (src/lxml/lxml.etree.c:37027)
  File "/usr/lib/python2.6/dist-packages/lxml/_elementpath.py", line 225, in findall
    return list(iterfind(elem, path))
  File "/usr/lib/python2.6/dist-packages/lxml/_elementpath.py", line 200, in iterfind
    selector = _build_path_iterator(path)
  File "/usr/lib/python2.6/dist-packages/lxml/_elementpath.py", line 184, in _build_path_iterator
    selector.append(ops[token[0]](_next, token))
KeyError: ':'

Solution 1

If root.nsmap contains the table namespace prefix then you could:

root.xpath('.//table:table', namespaces=root.nsmap)

findall(path) accepts {namespace}name syntax instead of namespace:name. Therefore path should be preprocessed using namespace dictionary to the {namespace}name form before passing it to findall().

Solution 2

Maybe the first thing to notice is that the namespaces are defined at Element level, not Document level.

Most often though, all namespaces are declared in the document's root element (office:document-content here), which saves us parsing it all to collect inner xmlns scopes.

Then an element nsmap includes :

• a default namespace, with None prefix (not always)
• all ancestors namespaces, unless overridden.

If, as ChrisR mentionned, the default namespace is not supported, you can use a dict comprehension to filter it out in a more compact expression.

You have a slightly different syntax for xpath and ElementPath.

---

So here's the code you could use to get all your first table's rows (tested with: lxml=3.4.2) :

import zipfile
from lxml import etree

# Open and parse the document
zf = zipfile.ZipFile('spreadsheet.ods')
tree = etree.parse(zf.open('content.xml'))

# Get the root element
root = tree.getroot()

# get its namespace map, excluding default namespace
nsmap = {k:v for k,v in root.nsmap.iteritems() if k}

# use defined prefixes to access elements
table = tree.find('.//table:table', nsmap)
rows = table.findall('table:table-row', nsmap)

# or, if xpath is needed:
table = tree.xpath('//table:table', namespaces=nsmap)[0]
rows = table.xpath('table:table-row', namespaces=nsmap)

Solution 3

Here's a way to get all the namespaces in the XML document (and supposing there's no prefix conflict).

I use this when parsing XML documents where I do know in advance what the namespace URLs are, and only the prefix.

        doc = etree.XML(XML_string)

        # Getting all the name spaces.
        nsmap = {}
        for ns in doc.xpath('//namespace::*'):
            if ns[0]: # Removes the None namespace, neither needed nor supported.
                nsmap[ns[0]] = ns[1]
        doc.xpath('//prefix:element', namespaces=nsmap)

Solution 4

Etree won't find namespaced elements if there are no xmlns definitions in the XML file. For instance:

import lxml.etree as etree

xml_doc = '<ns:root><ns:child></ns:child></ns:root>'

tree = etree.fromstring(xml_doc)

# finds nothing:
tree.find('.//ns:root', {'ns': 'foo'})
tree.find('.//{foo}root', {'ns': 'foo'})
tree.find('.//ns:root')
tree.find('.//ns:root')

Sometimes that is the data you are given. So, what can you do when there is no namespace?

My solution: add one.

import lxml.etree as etree

xml_doc = '<ns:root><ns:child></ns:child></ns:root>'
xml_doc_with_ns = '<ROOT xmlns:ns="foo">%s</ROOT>' % xml_doc

tree = etree.fromstring(xml_doc_with_ns)

# finds what you're looking for:
tree.find('.//{foo}root')

View more solutions