HttpClient: Only one usage of each socket address (protocol/network address/port) is normally permitted

Question

using (var client = new HttpClient())
{
 client.BaseAddress = new Uri(Url);
 client.DefaultRequestHeaders.Accept.Clear();
 client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));

 using (var task = client.PostAsJsonAsync(Url, body))
 {
    if (task.Result.StatusCode != HttpStatusCode.OK)
       throw new Exception(task.Result.ReasonPhrase);
 }

}

Not Sure why we get the Only one usage of each socket address (protocol/network address/port) is normally permitted xx.xxx.xxx.xx:80 error

System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted xx.xx.xx.xx:80
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

Solution 1

The error in question is WSAEADDRINUSE (10048):

Address already in use.
Typically, only one usage of each socket address (protocol/IP address/port) is permitted. This error occurs if an application attempts to bind a socket to an IP address/port that has already been used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). Client applications usually need not call bind at all—connect chooses an unused port automatically. When bind is called with a wildcard address (involving ADDR_ANY), a WSAEADDRINUSE error could be delayed until the specific address is committed. This could happen with a call to another function later, including connect, listen, WSAConnect, or WSAJoinLeaf.

Which means you either have multiple HttpClient objects trying to bind themselves to the same local IP/Port at the same time, or another app is using an IP/Port that an HttpClient is trying to also use.

More likely, you are probably posting HTTP requests too often, and maybe not fully consuming the responses, which would prevent ASP from pooling and reusing connections and thus encountering port exhaustion over time.

Solution 2

I hit the same in my load generator tool which tried to send 200 requests/sec to one server.

I moved from new HttpClient instance per request to a singleton and it did address it.

One note - initially I hit 2 requests/sec bottleneck but setting DefaultConnectionLimit to 500 solved it:

ServicePointManager.DefaultConnectionLimit = 500;

Solution 3

Recently ran into this problem on an existing client / server web services setup which has been working for years on Windows 2012 R2 environment.

• The issue would only appear after a long up time.
• The issue occurred on the outgoing client connection, not the listening socket on server.
• Using netstat we could see there was more than enough available sockets.
• Rebooting the client service did not solve the issue.
• Rebooting the app pool / iis reset would not solve the issue.
• Rebooting the server would resolve the issue for a period of time.

After exhausting all checks as suggested by posts and MS articles and combined knowledge of socket connections the call was made that this is a lower level code issue and we engaged Microsoft support.

The outcome was that there is a known issue for KB 4338824 https://support.microsoft.com/en-us/help/4338824/windows-81-update-kb4338824

I paraphrase.... "Applications and services may hang in closesocket() calls and cause applications or services to fail binding to network sockets."

The following patch was applied and the issue was resolved. https://support.microsoft.com/en-us/help/4345424/improvements-and-fixes-windows-8-1-and-server-2012-r2

Solution 4

Run command prompt as administrator and type following command. netstat -ano | findstr ":80"

• a: Displays all connections and listening ports.
• n: Displays addresses and port numbers in numerical form.
• o: Displays the owning process ID associated with each connection.

In case if you are using a different port number then replace 80 with the appropriate port number.

The result would show a process id using the 80 port number in the last column. Either you can change the port number in your program or you can kill the process using 80 port number.

View more solutions

Comments

[Tony Bao]

Thanks. It makes sense. I have multiple HttpClient instances in different threads, how one HttpClient instance know which ip/port being used by the other HttpClient instance in other thread?

[Tony Bao]

Thanks again, Remy. Now I understand why I got this error.

[Tony Bao]

From Henrik Nielsen: It's typically because you run out of ephemeral ports. Try and reuse the same HttpClient instance for all your requests

[Dibzmania]

@TonyBao So if the TCP connection is between the same client and server always, does reusing the same HttpClient means that the server always sees the same originating address and port ?

[Remy Lebeau]

@Dibzmania it depends on whether the HttpClient reuses the same TCP connection, or if the TCP connection is closed and then a new TCP connection uses the same local IP/port

[Remy Lebeau]

@Dibzmania HTTP is stateless, so it depends on whether the HttpClient reuses the same TCP connection between requests, or if it closes the TCP connection and then creates a new TCP connection that uses the same local IP/port

[Dibzmania]

So If i have a client/server setup where the client kind of fires like 40+ requests per second using the same HttpClient object, would I hit a concurrency bottleneck somewhere (as opposed to using a new HttpClient object for each request). Please note I am getting the error as the OP and as from the answer, reusing the same HttpClient would solve the issue (along with certain other changes)