Apache keepalive timeout

Question

I have a server with 2 cores and 2GB RAM which is struggling with memory utilisation.

When I run free -m I regularly get under 50MB free - right now it shows 378MB free with only 1 user actually on the site:

             total       used       free     shared    buffers     cached
Mem:          1869       1772         96          5         34        247
-/+ buffers/cache:       1491        378
Swap:         1023         34        989

When I run ps-aux I see lots of httpd requests which started yesterday and are still open.

apache    5789  0.0  3.8 1038176 72904 ?       S    03:18   0:27 /usr/sbin/httpd
apache    5790  0.0  3.3 1029400 64216 ?       S    03:18   0:26 /usr/sbin/httpd
apache    5792  0.1  3.4 1031124 65876 ?       S    03:18   0:35 /usr/sbin/httpd
apache   28042  0.1  3.3 1029648 64868 ?       S    04:53   0:28 /usr/sbin/httpd
apache   28043  0.0  3.8 1038412 73420 ?       S    04:53   0:23 /usr/sbin/httpd
apache   31728  0.1  3.8 1038516 73008 ?       S    07:53   0:14 /usr/sbin/httpd
root     31912  0.0  0.7 968160 14860 ?        Ss    2014   0:03 /usr/sbin/httpd
apache   31914  0.0  0.0 246808  1364 ?        S     2014   0:01 /usr/sbin/httpd
apache   31915  0.1  3.7 1038644 71700 ?       S     2014   1:02 /usr/sbin/httpd
apache   31916  0.1  3.7 1038480 71844 ?       S     2014   0:56 /usr/sbin/httpd
apache   31917  0.0  3.1 1026120 60044 ?       S     2014   0:52 /usr/sbin/httpd
apache   31920  0.1  4.1 1045972 79940 ?       S     2014   0:59 /usr/sbin/httpd
apache   31921  0.0  3.8 1039180 73040 ?       S     2014   0:50 /usr/sbin/httpd
apache   31923  0.1  3.3 1029684 63908 ?       S     2014   0:59 /usr/sbin/httpd
apache   31924  0.1  3.2 1027808 61632 ?       S     2014   0:53 /usr/sbin/httpd
apache   31927  0.1  3.2 1027844 61664 ?       S     2014   0:54 /usr/sbin/httpd
apache   31929  0.1  3.8 1039660 73528 ?       S     2014   0:59 /usr/sbin/httpd
apache   31931  0.1  3.2 1028592 62492 ?       S     2014   0:58 /usr/sbin/httpd
apache   31933  0.0  3.2 1029036 62876 ?       S     2014   0:52 /usr/sbin/httpd
apache   31934  0.1  3.3 1029568 63232 ?       S     2014   0:54 /usr/sbin/httpd
apache   32022  0.0  3.8 1038928 73104 ?       S     2014   0:41 /usr/sbin/httpd

Output of top sorted by memory usage is:

31920 apache    20   0 1021m  78m 9064 S  0.0  4.2   0:59.48 httpd
31929 apache    20   0 1015m  71m 8428 S  0.0  3.8   1:02.44 httpd
28043 apache    20   0 1014m  71m 9176 S  0.0  3.8   0:26.59 httpd
32022 apache    20   0 1014m  71m 8796 S  0.0  3.8   0:44.04 httpd
31921 apache    20   0 1014m  71m 8408 S  0.0  3.8   0:51.95 httpd
31728 apache    20   0 1014m  71m 8768 S  0.0  3.8   0:15.00 httpd
 5789 apache    20   0 1013m  71m 8716 S  0.0  3.8   0:28.57 httpd
  591 apache    20   0 1013m  71m 8756 S  0.0  3.8   0:42.92 httpd
31916 apache    20   0 1014m  70m 8368 S  0.0  3.8   0:59.94 httpd
31915 apache    20   0 1014m  70m 8384 S  0.0  3.7   1:04.74 httpd
 5792 apache    20   0 1006m  64m 8744 S  0.0  3.4   0:37.37 httpd
 5790 apache    20   0 1006m  63m 8800 S  0.0  3.4   0:28.23 httpd
28042 apache    20   0 1005m  63m 9176 S  0.0  3.4   0:29.50 httpd
  584 apache    20   0 1006m  62m 8456 S  0.0  3.4   0:37.14 httpd
  586 apache    20   0 1006m  62m 8412 S  0.0  3.4   0:44.48 httpd
  588 apache    20   0 1006m  62m 8424 S  0.0  3.4   0:43.11 httpd
  587 apache    20   0 1006m  62m 8420 S  0.0  3.4   0:47.20 httpd
31923 apache    20   0 1005m  62m 8796 S  0.0  3.3   1:01.21 httpd
31933 apache    20   0 1005m  62m 8376 S  0.0  3.3   0:54.89 httpd
31927 apache    20   0 1004m  60m 8392 S  0.0  3.2   0:58.68 httpd
31924 apache    20   0 1003m  60m 8412 S  0.0  3.2   0:53.68 httpd
31934 apache    20   0 1003m  60m 8408 S  0.0  3.2   0:56.56 httpd
31917 apache    20   0 1003m  60m 8404 S  0.0  3.2   0:54.97 httpd
 1871 root      20   0  157m  17m 1968 S  0.0  0.9   0:53.10 lfd
 5597 apache    20   0  952m  16m 5212 S  0.0  0.9   0:01.02 httpd
31912 root      20   0  945m  14m  13m S  0.0  0.8   0:03.37 httpd
 5854 root      20   0 98364 4676 3672 S  0.0  0.2   0:00.09 sshd
 6557 postfix   20   0 82264 4420 3292 S  0.0  0.2   0:00.04 cleanup
 6560 postfix   20   0 82168 4316 3324 S  0.0  0.2   0:00.03 smtp
 4411 postfix   20   0 81964 4048 3096 S  0.0  0.2   0:00.04 pickup
 5856 root      20   0  105m 1944 1528 S  0.0  0.1   0:00.02 bash

In my httpd.conf I have:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 2


<IfModule prefork.c>
StartServers       4
MinSpareServers    12
MaxSpareServers   24
ServerLimit      50
MaxClients       50
MaxRequestsPerChild  3000
</IfModule>

From the above I assume this means that KeepAlive is on but that it will only keep any session alive for 2 seconds - if that is correct, why do I still have lots of processes running from yesterday and how can I remediate against this?

Solution

To confirm that Apache is taking all your memory, you can use this snipped:

function memperuser() {
ps aux | awk 'NR != 1 {x[$1] += $4} END{ for(z in x) {print z, x[z]"%"}}'
}

Then type memperuser.

Another useful way to see the shared and private memory break-down is to use ps_mem.py

That will show you how much shared and private memory Apache is consuming.

You should also take a look through all of the LoadModule entries in your httpd.conf and determine if you need all of them. Usually the biggest culprit is mod_php. If you don't need PHP, then simply disable that module. It might get loaded in its own config under httpd.d/

If you are not using PHP, you can also reduce the memory footprint of Apache by using mpm_event or mpm_worker instead of mpm_prefork. The Apache website has some really good documentation on how to use the different MPM modules.

You can get even further reduction of memory footprint by using the latest APR apr, apr-util and apr-iconv and Apache 2.4 branch. How you go about compiling and/or installing that is highly dependent on your setup, so I will defer to your specific OS forums.

All of these options really depend on how you are using Apache.