A fast python HTML parser

Question

I wrote a python script that processes a large amount of downloaded webpages HTML(120K pages). I need to parse them and extract some information from there. I tried using BeautifulSoup, which is easy and intuitive, but it seems to run super slowly. As this is something that will have to run routinely on a weak machine (on amazon) speed is important. is there an HTML/XML parser in python that will work much faster than BeautifulSoup? or must I resort to regex parsing..

Solution 1

lxml is a fast xml and html parser: http://lxml.de/parsing.html

Solution 2

Streaming (or SAX-style) parsers can be faster than DOM-style ones. Your code is passed elements one at a time as they occur in the document, and although you have to infer (and keep track of) their relationships yourself, you only need to maintain as much state as is required to locate the data you want. As a bonus, once you've found what you're interested in, you can terminate parsing early, saving the time that would have been required to process the rest of the document.

In contrast, DOM-style parsers need to build a complete navigable object model of the whole document, which takes time (and memory). DOM-style parsers are typically built on top of streaming parsers, so they will ceteris paribus be slower than the streaming parser they use.

Python has a streaming parser for HTML called html.parser. Depending on how hard it is to recognize the data you want to extract, it can be complicated to actually program a streaming parser to do scraping, because the API is sort of inside-out from the way you're used to thinking of documents. So it may be worth choosing an easier-to-use parser even if it's slower at runtime, because simple code that works is generally better than complicated code with bugs.

On the gripping hand, a parser written in C (such as lxml) is going to blow the doors off pretty much any parser written in pure Python, regardless of what approach it takes, so that might be a way to get the speed you need. (In fact, these days, BeautifulSoup uses lxml as its default parser.)

Comments

[Admin]

Keep the pony away...

[Admin]

I have no experience with parsing HTML in Python, but here are some benchmark results that you may find useful.

[Admin]

regex and HTML == failure

[WeaselFox]

@JackManey - wow. I will definitely not parse HTML with regex after this...

[inspectorG4dget]

I was going to suggest this as well... although, I don't have any data to support performance ratios of this against BeautifulSoup

[WeaselFox]

thanks, the benchmarking does show that lxml is much faster!

[Flimm]

To install: pip install lxml

[jvmvik]

A benchmark is available here. medium.com/@vikoky/…