HttpClient throwing "An error occurred while sending the request."

Question

I have three layer application architecture.

My Client --> My service A (REST hosted in IIS) --> Other Team's service X (REST).

Service A is ASP.Net 4.6.1 framework, not ASP.Net Core.

Client is communicating to A with HttpClient and A is communicating to X with HttpClient.

Client is firing almost 2500 calls to my service to A and to X.

Out of 2500 calls service A randomly (may be 10 calls) fails with below exception. Its not reproducible.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> 
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a 
receive. ---> System.IO.IOException: Unable to read data from the transport connection: An     
established connection was aborted by the software in your host machine. ---> 
System.Net.Sockets.SocketException: An established connection was aborted by the software in your 
host machine
at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags 
socketFlags, AsyncCallback callback, Object state)
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback 
callback, Object state)
 --- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace --

Here is my service A call. A in IIS calls below code block and its called by each request. X is taking user credentials and returning data based on user, so we are not sharing HttpClient between calls.

var user = (System.Security.Principal.WindowsIdentity)HttpContext.Current.User.Identity;
            System.Security.Principal.WindowsIdentity.RunImpersonated(user.AccessToken, () =>
      {
        static HttpClient Client = new HttpClient();
        static string CallX(string[] args)
        {
            HttpClientHandler handler = new HttpClientHandler
            {
                UseDefaultCredentials = true
            };    

            Client = new HttpClient(handler)
            {
                BaseAddress = new Uri("http://XserviceUrl/api/")
            };
            Client.Timeout = TimeSpan.FromSeconds(600);    
            var result = Client.PostAsync("Fake X controller"
                , new StringContent(JsonConvert.SerializeObject(args)
                , Encoding.UTF8, "application/json")).Result;    

            result.EnsureSuccessStatusCode();

            var json = result.Content.ReadAsStringAsync().Result;
            return DosomethingWithResult(json);    
        }
    });

Things I tried:

Some SO post suggested might be timeout issue. So I added 600 seconds in Client and in Service A. I also changed IIS request timeout from default 2 minutes to 10 (600 seconds).

Solution 1

After some digging I resolved the issue. When A was sending the request to X, A was setting the Connection: keep-alive and the X was responding with Connection: Close property in header.

So after some calls A was exhaust with opened tcp connection and randomly it was throwing the error.

(Fiddler helped me to figure this out)

So all I had to do was setting ConnectionClose property of HttpClient

_client.DefaultRequestHeaders.ConnectionClose = true;

Solution 2

I face the exact same problem in production environment. Similar setup and about 30k http calls send from the client. Errors occur in a very infrequent way and it is very difficult to reproduce.

After reading numerous posts I think this Microsoft's bug when doing the connection pooling in HttpClient (in my solution I use httpClientFactory). You can take a look at https://github.com/dotnet/runtime/issues/26629

The approach followed to overcome this issue until Microsoft fixed the issue:

1. Retry policy. Polly has been used and when this exception occurs the call is retried after a couple of seconds.

2. Increased the request timeout to 3 minutes.

It seems to work fine for now but as I mentioned before it is hard to reproduce the error in a controlled way.

Comments

[user781700]

My service is ASP.Net Framework so I am not sure how can I use httpClientFactory

[Stelios Giakoumidis]

It does not matter. I suppose that you use a static HttpClient. If you set up the timeout to 3 minutes and you use Polly github.com/App-vNext/Polly/wiki/Retry That solution is tested and worked out fine for me

[Stelios Giakoumidis]

@user781700 did you try the retry policy and increased timeouts? I have actually just finished some load tests, and it works very well under quite high pressure. Let me know if you have other issues

[user781700]

I do not think increasing the timeout is the problem here. All the request finishes in less than 1 second. Before I introduce new API (Polly), I wanted to know why is this happening.

[Stelios Giakoumidis]

@user781700 the reason we increase the timeout is so the bug in connection pooling will have enough time to pop up. After increasing timeout period you will see that in about 2 minutes time you will get have a retry attempt. If you try to catch the error you will see that this is TaskCanceledException (timeout). This relates to the bug in httpclient.

[Adriano Tumminelli]

This worked for me, i was getting that error with this code var httpMessage = await client.PostAsync(filenameURL, formContent);