Parse XML in Haskell

Question

I'm trying to get data from a webpage that serves a XML file periodically with stock market quotes (sample data). The structure of the XML is very simple, and is something like this:

<?xml version="1.0"?>
<Contents>
  <StockQuote Symbol="PETR3" Date="21-12-2010" Time="13:20" Price="23.02" />
</Contents>

(it's more than that but this suffices as an example).

I'd like to parse it to a data structure:

 data Quote = Quote { symbol :: String, 
                      date   :: Data.Time.Calendar.Day, 
                      time   :: Data.Time.LocalTime.TimeOfDay,
                      price  :: Float}

I understand more or less how Parsec works (on the level of the Real World Haskell book), and I tried a bit the Text.XML library but all I could develop was a code that worked but is too big for such a simple task and looks like a half baked hack and not the best one could do.

I don't know a lot about parsers and XML (I know basically what I read in the RWH book, I never used parsers before) (I just do statistical and numerical programming, I'm not a computer scientist). Is there a XML parsing library where I could just tell what is the model and extract the information right away, without having to parse each element by hand and without having to parse pure string?

I'm thinking about something like:

  myParser = do cont  <- openXMLElem "Contents"
                quote <- openXMLElem "StockQuote" 
                symb <- getXMLElemField "Symbol"
                date <- getXMLElemField "Date"
                (...) 
                closequote <- closeXMLElem "StockQuote"
                closecont  <- closeXMLElem "Contents"
                return (symb, date)


  results = parse myParser "" myXMLString

where I wouldn't have to deal with the pure string and create the combinators myself (I suck at it).

EDIT: I probably need to read a bit (just enough to get this done the right way) about parsers in general (not only Parsec) and the minimum about XML. Do you guys recomend something?

The real string I have to parse is this:

 stringTest = "<?xml version=\"1.0\"?>\r\n<ComportamentoPapeis><Papel Codigo=\"PETR3\" 
 Nome=\"PETROBRAS ON\" Ibovespa=\"#\" Data=\"05/01/201100:00:00\" 
 Abertura=\"29,80\" Minimo=\"30,31\" Maximo=\"30,67\" Medio=\"30,36\" 
 Ultimo=\"30,45\" Oscilacao=\"1,89\" Minino=\"29,71\"/></ComportamentoPapeis>\r\n"

EDIT2:

I tried the following (readFloat, readQuoteTime, etc... are just functions to read things from strings).

bvspaParser :: (ArrowXml a) => a XmlTree Quote
bvspaParser = hasName "ComportamentoPapeis" /> hasName "Papel" >>> proc x -> do
   (hour,date) <- readQuoteTime ^<< getAttrValue "Data" -< x
   quoteCode   <- getAttrValue "Codigo" -< x
   openPrice   <- readFloat ^<< getAttrValue "Abertura" -< x
   minim       <- readFloat ^<< getAttrValue "Minimo" -< x
   maxim       <- readFloat ^<< getAttrValue "Maximo" -< x
   ultimo      <- readFloat ^<< getAttrValue "Ultimo" -< x
   returnA     -< Quote quoteCode (LocalTime date hour) openPrice minim maxim ultimo

docParser :: String -> IO [Quote]
docParser  str = runX $ readString [] str >>> (parseXmlDocument False) >>> bvspaParser

When I call it in ghci:

*Main> docParser stringTest >>= print
[]

Is anything wrong?

Solution 1

I've used Haskell XML Toolbox in the past. Something along the lines of

{-# LANGUAGE Arrows #-}

quoteParser :: (ArrowXml a) => a XmlTree Quote
quoteParser =
    hasName "Contents" /> hasName "StockQuote" >>> proc x -> do
    symbol <- getAttrValue "Symbol" -< x
    date <- readTime defaultTimeLocale "%d-%m-%Y" ^<< getAttrValue "Date" -< x
    time <- readTime defaultTimeLocale "%H:%M" ^<< getAttrValue "Time" -< x
    price <- read ^<< getAttrValue "Price" -< x
    returnA -< Quote symbol date time price

parseQuoteDocument :: String -> IO (Maybe Quote)
parseQuoteDocument xml =
    liftM listToMaybe . runX . single $
    readString [] xml >>> getChildren >>> quoteParser

Solution 2

There are plenty of XML libraries written for Haskell that can do the parsing for you. I recommend the library called xml (see http://hackage.haskell.org/package/xml). With it, you can simply write e.g.:

let contents = parseXML source
    quotes   = concatMap (findElements $ simpleName "StockQuote") (onlyElems contents)
    symbols  = map (findAttr $ simpleName "Symbol") quotes
    simpleName s = QName s Nothing Nothing
print symbols

This snippet prints [Just "PETR3"] as a result for your example XML, and it's easy to extend for collecting all the data you need. To write the program in the style you describe you should use the Maybe monad, as the xml lookup functions often return a Maybe String, signaling whether the tag, element or attribute could be found. Also see a related question: Which Haskell XML library to use?

Solution 3

For simple xml parsing, you can't go wrong with tagsoup. http://hackage.haskell.org/package/tagsoup

Solution 4

The following snippet uses xml-enumerator. It leaves date and time as text (parsing those is left as an exercise to the reader):

{-# LANGUAGE OverloadedStrings #-}
import Text.XML.Enumerator.Parse
import Data.Text.Lazy (Text, unpack)

data Quote = Quote { symbol :: Text
                   , date   :: Text
                   , time   :: Text
                   , price  :: Float}
  deriving Show

main = parseFile_ "test.xml" (const Nothing) $ parseContents

parseContents = force "Missing Contents" $ tag'' "Contents" parseStockQuote
parseStockQuote = force "Missing StockQuote" $ flip (tag' "StockQuote") return $ do
    s <- requireAttr "Symbol"
    d <- requireAttr "Date"
    t <- requireAttr "Time"
    p <- requireAttr "Price"
    return $ Quote s d t (read $ unpack p)

Solution 5

There are other ways to use this library, but for something simple like this I threw together a sax parser.

import Prelude as P
import Text.XML.Expat.SAX
import Data.ByteString.Lazy as L

parsexml txt = parse defaultParseOptions txt :: [SAXEvent String String]

main = do
  xml <- L.readFile "stockinfo.xml"
  return  $ P.filter stockquoteelement (parsexml xml)

  where
    stockquoteelement (StartElement "StockQuote" attrs) = True
    stockquoteelement _ = False

From there you can figure out where to go. You could also use Text.XML.Expat.Annotated in order to parse it into a structure that is more like what you are looking for above:

parsexml txt = parse defaultParseOptions txt :: (LNode String String, Maybe XMLParseError)

And then use Text.XML.Expat.Proc to surf the structure.

View more solutions

Comments

[Michael Snoyman]

As long as you don't need to validate well-formedness or ensure that tags are well balanced. As much as I like tagsoup for HTML scraping, I think it's ill-suited for parsing well structured XML files.

[sclv]

@Michael -- if i'm parsing someone else's irritating format, I generally don't care if they've got the details right, or I trust them to have done so or not depending on the competency of the vendor. I care about getting my information out, and robustly so if they go changing things on me.

[Rafael S. Calsaverini]

This is nice. I like arrows. But I can't find anyway to get a String and return an XmlTree to feed the parser. I only find functions to read documents. Is there any (ArrowXml a) => a String XmlTree function?

[Rafael S. Calsaverini]

I'm having a problem with the first line <?xml version=\"1.0\"?>. When it's present the parser can't get anythig. I solved this by simply droping 23 characters from the string. Is there a less hacky solution?

[MtnViewMark]

You shouldn't have to do that. I don't know HXT that much, but I think the issue is that the code above parses an XML fragment, not an XML document. Look at Text.XML.HXT.Arrow.ProcessDocument.

[Matthieu M.]

@ephemient: while nifty, all those operator overloads makes me feel like I'm reading perl... it's scary!

[Rafael S. Calsaverini]

@MtnViewMark : can you please take a look on my EDIT2 and see if you detect what is my mistake? I still can't do it without deleting the first characters from the string.

[ephemient]

@Rafael: You must descend from the root of the document to the (single) child you wish to parse. Or you can use deep to search the whole tree.

[Rafael S. Calsaverini]

Hum... thanks. I think I got it now. I'll have another look in the docs.