Tuning Apache2 prefork MaxClients ServerLimit

Question

I have a machine with 128 GB Ram that is using Apache2 as Webserver (in this machine there is no Database Server, the Database Machine is a 64 GB Ram machine that can handle 2000 max connections). I see with a monitoring tool that there are at the moment about 44 busy workers and 12 idle workers, what are best theoretical values for my prefork module?

i got blank pages sometimes loading websites on high load hours and got this error on my apache error log:

[notice] child pid 13595 exit signal Segmentation fault (11)

how can solve this issue too?

My Apache2 Prefork Module configuration:

StartServers          3
MinSpareServers       3
MaxSpareServers       5
ServerLimit           3200
MaxClients            3100
MaxRequestsPerChild   0

Free -h on the www Machine :

total: 128 G free: 97GB (with apache2 running) shared 0b buffers 1.9G cache 23G

Ram used By Apache2 and other Programs:

Private  +   Shared  =  RAM used    Program

 96.0 KiB +  61.0 KiB = 157.0 KiB   sh
176.0 KiB +  26.0 KiB = 202.0 KiB   atd
176.0 KiB +  35.5 KiB = 211.5 KiB   acpid
208.0 KiB +  19.5 KiB = 227.5 KiB   mdadm
204.0 KiB +  30.0 KiB = 234.0 KiB   init
248.0 KiB +  62.0 KiB = 310.0 KiB   sendmail
376.0 KiB +  36.0 KiB = 412.0 KiB   dbus-daemon
388.0 KiB + 285.5 KiB = 673.5 KiB   cron (2)
820.0 KiB +  42.0 KiB = 862.0 KiB   gam_server
920.0 KiB + 108.0 KiB =   1.0 MiB   ntpd
968.0 KiB + 243.0 KiB =   1.2 MiB   getty (6)
  1.3 MiB + 351.5 KiB =   1.6 MiB   udevd (3)
  1.5 MiB + 343.0 KiB =   1.8 MiB   sendmail-msp
  2.0 MiB + 910.0 KiB =   2.9 MiB   plugin-localresources2
  3.4 MiB +  50.0 KiB =   3.4 MiB   rsyslogd
  3.6 MiB +  68.5 KiB =   3.7 MiB   bash
  1.9 MiB +   2.1 MiB =   4.0 MiB   sendmail-mta (4)
  3.8 MiB + 556.0 KiB =   4.3 MiB   sshd (2)
  3.7 MiB +   1.2 MiB =   4.8 MiB   plugin-apache2
  5.1 MiB +   1.2 MiB =   6.3 MiB   agent-service
  7.0 MiB + 654.0 KiB =   7.6 MiB   fail2ban-server
  9.6 MiB +   2.6 MiB =  12.2 MiB   proftpd (8)
 59.2 MiB +  70.0 KiB =  59.3 MiB   miniserv.pl
 96.8 MiB +   3.6 MiB = 100.4 MiB   php5-cgi (2)
196.4 MiB +  35.9 MiB = 232.3 MiB   apache2 (40)
---------------------------------
                     tot 450.0 MiB

Solution

Apache prefork settings, per apache performance tuning guidelines

quote:

The single biggest hardware issue affecting webserver performance is RAM.
A webserver should never ever have to swap, as swapping increases the latency
of each request beyond a point that users consider "fast enough". 
This causes users to hit stop and reload, further increasing the load.
You can, and should, control the MaxClients setting so that your server does
not spawn so many children it starts swapping. This procedure for doing this
is simple: determine the size of your average Apache process, by looking at
your process list via a tool such as top, and divide this into your total 
available memory, leaving some room for other processes.

you should set it up like this based on your input to:

• Total Memory : 128 GB
• -10% memory for everything except apache: 115 GB
• Now we need to figure out how much single apache process is using.

To calculate this you can use following script :

pgrep apache2 | xargs -n1 -I{} cat /proc/{}/smaps | \
  awk '{if ($0 ~ /stack/) {pids+=1} else if ($0 ~/^Shared_/) 
    {shared+=$2} else if ($0 ~ /^Pss:/) {priv+=$2}} END {
      printf "%.2f MB\n",(priv+shared/(pids*pids))/1024}'

This is best estimate of how much single apache process is using memory while trying to proportionally dividing shared usage per number of active apache processes and adding it on top of Pss (proportional set size)

Finally you divide 115 GB with this figure and you get MaxClients/ServerLimit. From here you can relatively calculate other figures like

• StartServers 30% of MaxClients
• MinSpareServers 5% of MaxClients
• MaxSpareServers 10% of MaxClients
• ServerLimit == MaxClients
• MaxConnectionsPerChild 10000 (as conservative alternative to address possible problem with memory leaky apps)

Comments

[ravi yarlagadda]

What's the application code that's running within the web server? That's most likely the culprit.

[Hrvoje Špoljar]

please send few samples of apache2ctl status; is there anything in error_log maybe?

[User-N]

i hope somebody with more rep points than me will give you vote for this answer, thank you so much!

[Quinn Comendant]

Your calculation script gives me 842.13 MB. That's about an order of magnitude above what I would (apache 2.2 on CentOS 6.7).

[Hrvoje Špoljar]

@Quinn That is because it's not running in prefork mode. It's likely in worker mode.

[Quinn Comendant]

It's in prefork mode. Here's output from /server-info: i.imgur.com/SS2gIXI.png

[Hrvoje Špoljar]

@QuinnComendant could you run ps_mem.py (github.com/pixelb/ps_mem/blob/master/ps_mem.py) script and double check what it reports?

[Hrvoje Špoljar]

@QuinnComendant I checked and fact that prefork is in server-info does not mean it's running in prefork; it's simply compiled as prefork but configuration/setup is determined in runtime. Check here serverfault.com/questions/88000/…

[Quinn Comendant]

By all methods given in answers to that question (httpd -l, httpd -V, ps aux | grep worker, etc) all indicate that I'm running prefork. The output from the ps_mem.py script includes, 644.7 MiB + 324.8 MiB = 969.5 MiB httpd (34).

[Hrvoje Špoljar]

@QuinnComendant I'm running same setup on Centos6 with Apache 2.2 in worker mode and also all those suggest it is prefork but it's not... I'm looking for better way to check this out and when I find one I'll update answer

[Quinn Comendant]

@HrvojeŠpoljar How about adding this to the conf: <IfModule prefork.c> Header append X-MPM prefork </IfModule><IfModule worker.c> Header append X-MPM worker </IfModule>. When I do this, I get the header, X-MPM: prefork and I do not get the header X-MPM: worker.

[Hrvoje Špoljar]

@QuinnComendant wicked trick however IfModule will be true if prefork module is available (which it is), and actually both options are available worker and MPM... I have identical server and memory usage on my system for worker is ~850 MB. There is no way prefork can use for 1 process 800 MB of memory. You altered line in script where it references process name apache2 to httpd right ?

[shawn]

The script "(priv+shared/(pids*pids))/1024" is totally wrong.

[Hrvoje Špoljar]

@shawn why would you say so ? how do you believe it should be calculated?

[IanB]

The script is giving wildly inaccurate memory usage for me (Centos 6). Instead, I run top, sort by memory usage and take the average number under the RES column.

[Hrvoje Špoljar]

@IanB top is inaccurate, RES does not account for memory segments that are shared. Say you have 20 processes, each using 100 Mb of mem. Of each 100 Mb of mem each process uses not whole 100 Mb is used fully, some portion is shared so in reality they use less than 100. Top is old utility and it does not support detailed memory statistics which whould precisely account for shared portions of memory of group of processes like in apache prefork situation.