Read headers from HttpResponseMessage before Content is 100% complete

Question

1. How do I access the response headers, before the entire response has been streamed back?
2. How do I read the stream as it arrives?
3. Is HttpClient my best choice for such granular control of receiving the http response?

Here's a snip that might illustrate my question:

using (var response = await _httpClient.SendAsync(request,
  HttpCompletionOption.ResponseHeadersRead))
{
   var streamTask = response.Content.ReadAsStreamAsync();
   //how do I check if headers portion has completed? 
   //Does HttpCompletionOption.ResponseHeadersRead guarantee that?
   //pseudocode
   while (!(all headers have been received)) 
     //maybe await a Delay here to let Headers get fully populated
   access_headers_without_causing_entire_response_to_be_received

   //how do I access the response, without causing an await until contents downloaded?
   //pseudocode
   while (stremTask.Resul.?) //i.e. while something is still streaming
     //? what goes here? a chunk-read into a buffer? or line-by-line since it's http?
   ...

Edit to clarify another gray area for me:
Any reference I unearthed has some kind of a blocking statement, that would cause await for the contents to arrive. References I read usually access methods or properties on the streamTask.Result or on the Content, and I don't know enough to rule out which such references are okay as the streamTask is progressing vs. which are going to cause an await until the task completes.

Solution 1

Based on my own testing, the content won't be transferred until you start reading the content stream, and you're correct that calling Task.Result is a blocking call, but its very nature, it's a synchronisation point. But, it doesn't block to pre-buffer the entire content, it only blocks until the content starts to come from the server.

So an infinite stream won't block for an infinite amount of time. As such, trying to fetch the stream asynchronously might be deemed overkill, especially if your header processing operation is relatively short. But if you want to, you can always process the headers while the content stream is being handled on another task. Something like this would accomplish that.

static void Main(string[] args)
{
    var url = "http://somesite.com/bigdownloadfile.zip";
    var client = new HttpClient();
    var request = new HttpRequestMessage(HttpMethod.Get, url);

    var getTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
    Task contentDownloadTask = null;

    var continuation = getTask.ContinueWith((t) =>
    {
        contentDownloadTask = Task.Run(() =>
        {
            var resultStream = t.Result.Content.ReadAsStreamAsync().Result;
            resultStream.CopyTo(File.Create("output.dat"));
        });

        Console.WriteLine("Got {0} headers", t.Result.Headers.Count());
        Console.WriteLine("Blocking after fetching headers, press any key to continue...");
        Console.ReadKey(true);
    });

    continuation.Wait();
    contentDownloadTask.Wait();
    Console.WriteLine("Finished downloading {0} bytes", new FileInfo("output.dat").Length);

    Console.WriteLine("Finished, press any key to exit");
    Console.ReadKey(true);
}

Note that there's no need to check if the headers portion is complete, you've explicitly specified that with the HttpCompletionOption.ResponseHeadersRead option. The SendAsync task will not continue until the headers have been retrieved.

Solution 2

The result using the await/async keywords it's even more readable:

var url = "http://somesite.com/bigdownloadfile.zip";

using (var httpClient = new HttpClient())
using (var httpRequest = new HttpRequestMessage(HttpMethod.Get, url ))
using(HttpResponseMessage response = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead))
using (Stream stream = await response.Content.ReadAsStreamAsync())
{
    //Access to the Stream object as it comes, buffer it or do whatever you need
}    

Comments

[G. Stoynev]

How about reading the stream in chunks - buffer or strings/lines? Is HttpClient the most appropriate class for such a granular control of http download?

[Snixtor]

To read in chunks you could use Stream.Read - msdn.microsoft.com/en-us/library/system.io.stream.read.aspx - Though you'd need a fairly particular situation to justify it (it can be a bit clumsy and slow compared with CopyTo. If you're wanting to read line-by-line, wrap the stream in a StreamReader - msdn.microsoft.com/en-us/library/system.io.streamreader.aspx

[Snixtor]

As for HttpClient, once you've got the response content stream it's sort of out of the picture. It manages the request and response, headers, some error processing, etc. You're not going to get much more flexibility than having direct access to the response stream, at least not in managed code.