Stream from response.Content.ReadAsStreamAsync() is not readable randomly

Question

I am making a .Net Web API application in which the following code is making call to my different c# application to download file and then save it on the disk. Sometimes everything works fine and I get the file but sometimes the below code is not able to read the stream and I can see the remote connection closed exception in my other application.

public async Task<string> GetFilePathAsync(TestModel model)
{
    string filePath = string.Empty;
    var response = await cgRequestHelper.DownloadAsync(model);  

    if (response.IsSuccessStatusCode)
    {                    
        filePath = await SaveCgStreamAsync(cgResponse, serviceModel.FileName);
    }
    return filePath;
}

public async Task<HttpResponseMessage> DownloadAsync(TestModel model)
{            
    if (model == null)
        throw new ArgumentNullException("model");
    if (string.IsNullOrEmpty(model.Url))
        throw new ArgumentNullException("Url");
    if (model.Headers == null)
        throw new ArgumentNullException("Headers");

    HttpResponseMessage response;
    using (HttpClient httpClient = new HttpClient())
    {
        foreach (var header in model.Headers)
        {
            httpClient.DefaultRequestHeaders.Add(header.Key, header.Value);
        }
        response = await httpClient.GetAsync(model.Url, HttpCompletionOption.ResponseHeadersRead); 
    }
    return response;            
}        

public async Task<string> SaveCgStreamAsync(HttpResponseMessage response, string fileName)
{
    if (response == null)
        throw new ArgumentNullException("response");
    if (string.IsNullOrEmpty(fileName))
        throw new ArgumentNullException("fileName");

    var filePath = _CreateTemporaryLocation(fileName);
    Stream cgStream = null;
    Stream fileStream = null;
    try
    {
        cgStream =  await response.Content.ReadAsStreamAsync();
        fileStream = File.Open(filePath, FileMode.Create);
        await cgStream.CopyToAsync(fileStream);
    }
    catch(Exception e)
    {
        throw;
    }
    finally
    {
        if(cgStream != null)
            cgStream.Dispose();
        if(fileStream != null)
            fileStream.Dispose();
    }

    return filePath;            
}

I have set ServicePointManager.DefaultConnectionLimit = 1000 in Global.asax.cs

In my current application I am getting exception on "await cgStream.CopyToAsync(fileStream);" line when it tries to read the cgStream. The exception is "Cannot access a closed Stream." with null InnerException The other application exception is: System.Web.HttpException: The remote host closed the connection. The error code is 0x800703E3. at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect) at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush() at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)

On an average, 1 out of 10 request is failing with the above error. Since it is random and not always failing, it is very difficult to troubleshoot the problem.

The above code is written with the help from: http://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient

Any help related to this problem is appreciated!

Thanks

Solution

I figured out the problem in the code: The problem was that I was initializing HttpClient object in 'using' and using its response outside the using scope. This will dispose the HttpClient object, hence breaking the remote connection and that is why I am not able to stream the content. The random behavior was because we do not know when will the object gets disposed, sometimes it does not gets disposed before streaming and sometimes it does.