Apache spawning enormous number of httpd processes

Question

I am running a CentOS 6.7 machine (16 GB vRAM, 8 vCPUs) with a simple web server setup (Apache/2.2.15, PHP/5.3.3, MySQL/5.1.66), hosting an online shop with a moderate number of page impressions (about 2,000 ~ 4,000 per day).

The server ran smooth over the last 3 years without any necessity to change its configuration. Now, since last week - somehow overnight - Apache/HTTP recurrently becomes inaccessible after a short period of time (about 30 minutes). I checked some parameters and saw that there are a lot of httpd processes running. ps axf | grep httpd | wc shows something like this:

387    2344   18354

Whereas the load is not very exciting. top looks like this:

A very small number of httpd processes is released again from time to time, but the total number keeps almost constantly increasing. If I run a service httpd reload, the number of processes drops back to 0 and begins to increase again over the next couple of minutes to hours. After a while, Apache's log tells me:

[error] server reached MaxClients setting, consider raising the MaxClients setting

I did this and I also adjusted different further config params, but it did not help. No matter what value MaxClients and ServerLimit are set to, Apache will not stop to spawn new httpd processes up to these limits. After this, the website is not accessible anymore.

There was no increase in page hits according to AWStats. Furthermore, there has been not a single change in the PHP application. Apart from the breakdown blocking all requests after 30 minutes, the website performs fast as usual. As a dirty and temporary workaround Cron keeps reloading Apache half-hourly.

Prefork/Worker settings have been as follows over the last 3 years:

<IfModule prefork.c>
StartServers       4
MinSpareServers    5
MaxSpareServers   10
ServerLimit      128 
MaxClients       128 
MaxRequestsPerChild  600
</IfModule>

<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75 
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

I raised them up to:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   10
ServerLimit       640
MaxClients       640
MaxRequestsPerChild  1000
</IfModule>

<IfModule worker.c>
StartServers         4
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

After this change, Apache/HTTP became unavailable after about 1 hr.

How can this happen all of a sudden and what options do I have to investigate further where this strange behaviour originates from?

Solution

Fist, increasing the prefork parameters will only worsen your situation. So restore the previous configuration.

To rule out any prefork problem/misconfiguration, try to disable it. Does the situation change significantly? If so, you find your problem.

If not, the problem is related to external factors: for example, maybe some bot/script is targeting your website, opening many connections (with the relative httpd process). If this is the case, you should be able to see that from Apache log files.

If this is not sufficient, you can enable the mod_status module to show detailed stats. Then, issue apachectl fullstatus to access these statistics. See here and here for more details.

Comments

[brent]

Enable mod_status and check what the processes are doing: httpd.apache.org/docs/current/mod/mod_status.html

[GregL]

Sounds to me like you're being DoSed with a Slowloris attack.

[proximus]

Thanks for your suggestions. I came across a large number of open connections in "Sending Reply" (W) status with the help of mod_status. Turned out it was kind of a Slowloris attack, as suspected by @GregL. It would have been difficult to figure out just by looking at the log files, because the bot sent requests from numerous IP addresses and used a faked Googlebot User-Agent. In a first step I blocked them in iptables. In a second step I am going to evaluate possible ways to mitigate Slowloris attacks (e. g. mod_reqtimeout).