Parsing large XML files in Adobe Flex

Question

I am working on an Adobe Flex app, which needs to parse a relativley large XML file. ATM it is only 35MB, but in an ideal world would get much larger in the future. **Edit: I have no control over the XML file

I am essentially dropping it's contents right into an SQLITE database, so I could use the SimpleXML class to turn it into an object and then iterate through it, but I am worried that this would be a bad approach as the file gets larger. Am I being paranoid, or is there a better way of doing this?

Solution 1

You will definitely run into some performance issues parsing an XML file that large. Back in Flex 2 days we used SOAP for services and had one data call that pulled back about 5K records and the Flash Player would hang / browser go unresponsive for about 10 seconds on a reasonably fast machine. I can't remember the size of that SOAP message but it couldn't have been more than 1-2 MB.

If it's possible for your backend to transform the XML into an object graph and send it back over AMF you will see much better performance. Flash Player does really well with large datasets provided they're encoded in AMF (condensed binary format).

Even stil, I'd really consider whether you want to send a single result that large of break it up into pieces. At least that way you have a path for better scaling and can give the user some better feedback, i.e. displaying a message such as "Processing Item 6 of 35..."

Solution 2

What you want is a SAX XML parser - it can parse a stream without reading the whole thing. I can't find one for AS3, though (although there are other people looking for the same thing.)

SAX operates by raising events as elements traverse the input stream. Pretty handy - I've used it often in the past, and once you're familiar with it, it's useful for a lot of cases. It even works with an open socket where the stream never closes.

Solution 3

As others have said, parsing large amounts of XML is not suggested and can get quite sluggish. It has been stated that the fastest way to send data between the flash client and a server side script is with AMF (Action Message Format) which is binary. If you have ever done anything with the SharedObject class then you have already had some dealing with AMF, as this is the format it writes the LSO to your hard drive as. AMFPHP was the best solution for this until recently as it now lends itself to the Zend framework, more specifically now it is ZendAMF.

There is an excellent tutorial here, by Lee Brimelow, one of the flash developers I look to for inspiration and clarity, that shows how to use ZendAMF.

The rate at which your data is available with ZendAMF, compared to plain old XML is staggering, and the larger the data to be parsed, the more noticeable.

Solution 4

Personally I would try to avoid XML files of this size. It's one of the cons of XML, you need to read the whole file before you can start using it.

Can't you have your database return smaller XML portions, with what you need, instead of everything at once? It might be a bit more difficult to set up but it the end might prove much more scalable.

View more solutions

Comments

[Dan Rosenstark]

Good point. But I'm even having trouble handling big XML files in AIR! stackoverflow.com/questions/1159154/…

[1owk3y]

Having the same problem (yes I'm aware Flash is horrifyingly antiquated. This doesn't align with business realities). I'm also going to need an intermediary parser to reduce the size of the payload. Very frustrating that I can't introduce my own stream parser to optimise the parse (like dkretz mentioned)