Understanding this error: apr_socket_recv: Connection reset by peer (104)

Question

So, if I do some benchmarking with apache benchmark (ab), and I use large numbers of requests. Then sometimes in the middle of a test I get this error.

I don't even know what it means. So how can I fix it? Or is it just something that will happen if the server gets too many hits anyway? The problem is, if I run 10,000 hits, it'll all run perfectly. If I run it again, it'll get to 4000 and get the error:

apr_socket_recv: Connection reset by peer (104)

A little about my setup: I have nginx taking static requests and processing dynamic ones to apache. The file in question is served from cache by nginx, so I guess it's probably got to do with how nginx is handling the requests?

Ideas?

Solution 1

The error means that the other end (webserver) suddenly disconnected in the middle of the session. have a look at the apache or nginx error logs to see if there is anything suspicious there.

Solution 2

It means that server is heavly loaded with the request i.e, all the threads are busy serving the request. Solution : either increase the maxThread attribute count for connector in server.xml file or increase acceptCount attribute value.

acceptcount : The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused.

Solution 3

Besides the answers here, I have read a lot of other ones:

• Replace localhost by 127.0.0.1
• Update the apache version (I have ApacheBench, Version 2.3 <$Revision: 1807734 $>)
• Add -r (Then I get apr_pollset_poll: The timeout specified has expired (70007))

None of them helped.

I thought about switching to wrk after seeing similar struggles.

Finding the problem

The problem seems to be related to the amount of ephermal ports. I tried to set it from 50000 to 25000 as this is the port range. Still no luck. Then I got the impression it is related to TIME_WAIT and this blog post. I think I could confirm that:

$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

    1 CLOSE_WAIT
    1 established)
    1 Foreign
    4 LISTEN
    8 SYN_SENT
   62 SYN_RECV
  351 ESTABLISHED
13916 TIME_WAIT

What I tried

I didn't fix it so far :-/

According to sudo sysctl -a | grep net.ipv4.tcp, I have:

net.ipv4.tcp_tw_reuse = 0    # No luck setting only that to 1
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_fin_timeout = 60  # Setting it to 5 didn't help either