Too Many ESTABLISHED connection from a single IP address in Apache

Question

netstat -ntp |grep 80 shows too many ESTABLISHED connection from single IP address. Around 300 of them and it is not an attack and user is using a 2G connection to access Apache. This is the case with other 2G connections also. As a result of this Apache is running out of children.

Earlier it was showing too many close_wait and after enabling tcp_tw_reuse and tcp_tw-recycle there is not much close_wait but the number of ESTABLISHED connections increased.

We are using Ubuntu 11.04 having 48 GB ram

keepalive On
keepalive timeout 10
max clients 800
max-request-perchild 4000
timeout 300

I have set syn_ack to 1 and syn_retries to 2.

On wifi there is no such issue. Connections are closing properly, but with 2G connections Apache is running out of children and too many ESTABLISHED connection.

also i have tried setting timeout from default 300 to 30,but since our project is image hosting for mobile phones,clients couldn't upload images properly as they are getting frequent time out.Also there were a lot of 408 messages so changed it to the default 300

Solution 1

tcp_tw_reuse and tcp_tw-recycle and tcp-fin_timeout to 30

The fin timeout helps here but reuse and reccyle? Why?

keepalive timeout 10

This is just silly. Even with dialup, this should be 3 or less.

timeout 300

Do you know what this does? This might be the default but it is way too high again.

You might try capturing some of the traffic using wireshark to see exactly why the connections are not closing.

Is using mod_reqtimeout an option

Only if the client is very badly broken and you're not bothered about providing a service to them.

should we move to ngnix server

It'll certainly handle slow connections much more easily, however you might want to use as a proxy (and you can selectively/transparently route particular subnets via this using iptables)

Solution 2

The problem is related to low speed clients. Best way to solve this is to use reverse proxy solution e.g. nginx, varnish or similar software if from of your apache. A good reverse proxy server can handle thousands of connections without problems.

Why is setting Nginx as a reverse proxy a good idea?

Solution 3

Have you tried dropping the Timeout directive to something much lower, such as 10 or 5?

Alternatively, you could try switching away from the prefork MPM (if possible) and use an event-driven model such as the event MPM in Apache 2.4 or a different web server such as nginx.

You could also use nginx (or similar) as a reverse proxy in front of Apache. The proxy will wait until it has received the entire request before making the upstream request to Apache. This request will then have no delays in the middle of it.

The problem with running out of Apache children can also be caused or exacerbated by the Keepalive settings you have. Consider switching that off or lowering the keepalive timeout value. Making it too low may make it useless, of course.

Comments

[David Schwartz]

How does your application make and use its connections?

[LINUX4U]

There are set of Server API that is written in php, the request to the server are in JSON format and response is also JSON,

[David Schwartz]

How frequent are the requests?

[user3904302]

You've already posted this question under a different username yesterday.

[LINUX4U]

we enabled tcp_tw_reuse and tcp_tw-recycle because there were too many TIME_WAIT in the netstat o/p after setting tcp_tw_reuse TIME_WAIT got reduced

[Sameer]

Reverse proxy is the way to solve the problem of slow clients.

[sartis]

close_wait is an abnormal state, appearing because of buggy application inside the webserver. Using reverse proxy will shorten lifetime of close_wait, reducing the symptom, but not solving the root cause.