C# - How to read a continuous stream of XML over HTTP

Question

I am trying to figure out the best way to consume a continuous stream of XML data from a service that is sending the data as a "constant" feed over HTTP.

I was considering using HttpWebRequest/Response but I am not sure how that will behave if the data just continuously streams.

Any thoughts?

Solution 1

I have done this before, not with XML, but with data that needed to be parsed for state changes for an application. HttpWebResponse.GetResponseStream() method worked fine for this. Make sure to call Close() on this stream when you are done. I suggest a finally block.

HttpWebRequest req;

try
{
    req = (HttpWebRequest)WebRequest.Create("http://www.example.com");
    Stream stream = req.GetResponseStream();

    byte[] data = new byte[4096];
    int read;
    while ((read = data.Read(data, 0, data.Length)) > 0)
    {
         Process(data, read);
    }
}
finally
{
    if (req != null)
        req.Close();
}

Or, alternatively:

HttpWebRequest req;

try
{
    req = (HttpWebRequest)WebRequest.Create("http://www.example.com");
    Stream stream = req.GetResponseStream();

    XmlTextReader reader = new XmlTextReader(stream);

    while (reader.Read())
    {
       switch (reader.NodeType) 
        {
         case XmlNodeType.Element:
           Console.Write("<{0}>", reader.Name);
           break;
         case XmlNodeType.Text:
           Console.Write(reader.Value);
           break;
         case XmlNodeType.CDATA:
           Console.Write("<![CDATA[{0}]]>", reader.Value);
           break;
         case XmlNodeType.ProcessingInstruction:
           Console.Write("<?{0} {1}?>", reader.Name, reader.Value);
           break;
         case XmlNodeType.Comment:
           Console.Write("<!--{0}-->", reader.Value);
           break;
         case XmlNodeType.XmlDeclaration:
           Console.Write("<?xml version='1.0'?>");
           break;
         case XmlNodeType.Document:
           break;
         case XmlNodeType.DocumentType:
           Console.Write("<!DOCTYPE {0} [{1}]", reader.Name, reader.Value);
           break;
         case XmlNodeType.EntityReference:
           Console.Write(reader.Name);
           break;
         case XmlNodeType.EndElement:
           Console.Write("</{0}>", reader.Name);
           break;
       }       

    }       
}
finally
{
    if (req != null)
        req.Close();
}

Solution 2

Should be able to do it fairly easily. You'll need to get the response stream by calling Response.GetResponseStream() and then use the async ResponseStream.BeginRead() in a loop.

There is no Timeout setting on the Response but if you're consistently getting sent data then it should be fine.

Comments

[scherand]

I think there are some small corrections needed in order to make the code work. There is no HttpWebRequest.GetResponseStream() method. You need to get a HttpWebResponse object using HttpWebRequest.GetResponse() and call GetResponseStream() on that. Also in the finally block req is not to be used but the afore mentioned Response.

[esac]

agreed, i just typed it out from memory, not from an actual project.

[swestner]

How would this work with XML? The XML would be invalid until the whole xml doc was downloaded, which kind of defeats the purpose of streaming.