Is there any XPath processor for SAX model?

Question

I'm looking for an XPath evaluator that doesn't rebuild the whole DOM document to look for the nodes of a document: actually the object is to manage a large amount of XML data (ideally over 2Gb) with SAX model, which is very good for memory management, and give the possibility to search for nodes.

Thank you all for the support!

For all those who say it's not possible: I recently, after asked the question, found a project named "saxpath" (http://www.saxpath.org/), but I can't find any implementing project.

Solution 1

My current list (compiled from web search results and the other answers) is:

• http://code.google.com/p/xpath4sax/
• http://spex.sourceforge.net/
• https://github.com/santhosh-tekuri/jlibs/wiki/XMLDog (also contains a performance chart)
• http://www.cs.umd.edu/projects/xsq/ (uniersity project, dead since 10 years, GPL)
• MIT-Licensed approach http://softwareengineeringcorner.blogspot.com/2012/01/conveniently-processing-large-xml-files.html
• Other parsers/memory models supporting fast XPath:
  • http://vtd-xml.sourceforge.net/ ("The world's fastest XPath 1.0 implementation.")
  • http://jaxen.codehaus.org/ (contains http://www.saxpath.org/)
  • http://www.saxonica.com/documentation/sourcedocs/streaming/streamable-xpath.html

The next step is to use the examples of XMLDog and compare the performance of all these approaches. Then, the test cases should be extended to the supported XPath expressions.

Solution 2

We regularly parse 1GB+ complex XML files by using a SAX parser which extracts partial DOM trees that can be conveniently queried using XPath. I blogged about it here: http://softwareengineeringcorner.blogspot.com/2012/01/conveniently-processing-large-xml-files.html - Sources are available on github - MIT License.

Solution 3

XPath DOES work with SAX, and most XSLT processors (especially Saxon and Apache Xalan) do support executing XPath expressions inside XSLTs on a SAX stream without building the entire dom.

They manage to do this, very roughly, as follows :

1. Examining the XPath expressions they need to match
2. Receiving SAX events and testing if that node is needed or will be needed by one of the XPath expressions.
3. Ignoring the SAX event if it is of no use for the XPath expressions.
4. Buffering it if it's needed

How they buffer it is also very interesting, cause while some simply create DOM fragments here and there, others use very optimized tables for quick lookup and reduced memory consumption.

How much they manage to optimize largely depends on the kind of XPath queries they find. As the already posted Saxon documentation clearly explain, queries that move "up" and then traverse "horizontally" (sibling by sibling) the document obviously requires the entire document to be there, but most of them require just a few nodes to be kept into RAM at any moment.

I'm pretty sure of this because when I was still making every day webapp using Cocoon, we had the XSLT memory footprint problem each time we used a "//something" expression inside an XSLT, and quite often we had to rework XPath expressions to allow a better SAX optimization.

Solution 4

SAX is forward-only, while XPath queries can navigate the document in any direction (consider parent::, ancestor::, preceding:: and preceding-sibling:: axis). I don't see how this would be possible in general. The best approximation would be some sort of lazy-loading DOM, but depending on your queries this may or may not give you any benefit - there is always a worst-case query such as //*[. != preceding::*].

Solution 5

Sorry, a slightly late answer here - it seems that this is possible for a subset of XPath - in general it's very difficult due to the fact that XPath can match both forwards and backwards from the "current" point. I'm aware of two projects that solve it to some degree using state machines: http://spex.sourceforge.net & http://www.cs.umd.edu/projects/xsq. I haven't looked at them in detail but they seem to use a similar approach.

View more solutions