Apache 2.2 eventually using all memory (worker mpm)

7,232

Solution 1

KeepaliveTimeout would help reduce the hanging "read" workers.

MPM uses multiple processes with multiple threads to handle incoming traffic, so what you're seeing is normal Apache operation.

This default tuning may work better for your needs:

ServerLimit 12
StartServers 2
MaxClients 500
MinSpareThreads 25
MaxSpareThreads 50
ThreadsPerChild 50

This would cause a total of 10 processes to spawn (MaxClients / ThreadsPerChild) which would significantly reduce the memory footprint of your server. You can afford more threads across less processes because of CPU availability.

Note that changing these variables requires Apache to be fully stopped then restarted.

Solution 2

What's the relation between the processes shown in top with the worker servers? I was expecting to see at most 10 processes (as the ServerLimit in worker settings - not taking in account the main process).

That doesn't look like the top of an Apache server running the worker MPM. I'm running two servers, one with the worker MPM and another with the prefork MPM. With worker, the top lists the Apache processes as httpd.worker, and the number of processes match the number of active servers (lines of dots with some non-dot entries) visible via mod_status. With prefork, the processes are listed as httpd, and the number roughly matches the number of idle/active workers (each non-dot entry).

So given this mod_status output:

..................R.............................................
..............._RR__R_R__RC_RWC__...............................
R_WC__R___R_....................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................

top on a server running prefork should show roughly 31 httpd processes, and top on a server running worker should show 3 httpd.worker processes.

Share:
7,232

Related videos on Youtube

Daniel Costa
Author by

Daniel Costa

Updated on September 18, 2022

Comments

  • Daniel Costa
    Daniel Costa almost 2 years

    I'm tuning up a Apache 2.2 server (it's shared with MySQL + Courier + Mongo + Postgre). The server is bare metal (no virtualizataion) with 2 quad-core Xeon (total 16 core) and 12GB ECC RAM (upgrading to 24GB in the next days). MySql has innodb_buffer set to 2G and is doing ok. Mongo and Postgre are barely touched - just for development.

    The problem is that Apache is eating all memory available, eventually using all the swap (and therefore crashing MySql as it's using more memory). We have Apache 2.2 with worker MPM and mod_security. Whenever we restart it, we have aprox 8G of free RAM.

    I'm trying to understand a few things:

    • What's the relation between the processes shown in top with the worker servers? I was expecting to see at most 10 processes (as the ServerLimit in worker settings - not taking in account the main process).
    • Is the amount of RES memory used for each THREAD or for each PROCESS?
    • In the mod_status output below, there are lot of READING requests that stays there for a long time. I've tried enabling ExtendedStatus to discover what clients are there but I couldn't find any useful info - any suggestions?
    • with the actual memory usage, what amount of RAM would I need to serve 400 concurrent connections? (as the mod status shows 250 already depleted)
    • what about raising ThreadsPerChild instead of raising MaxClients/ServerLimit?
    <IfModule worker.c>
    StartServers         2
    MaxClients         250
    MinSpareThreads     25
    MaxSpareThreads     75
    ThreadsPerChild     25
    MaxRequestsPerChild  0
    ServerLimit 10
    </IfModule>
    

    TOP shows (only for apache):

    top - 16:30:21 up 46 days, 23:12,  2 users,  load average: 0.94, 0.97, 1.31
    Tasks: 460 total,   1 running, 459 sleeping,   0 stopped,   0 zombie
    Cpu(s):  2.8%us,  0.5%sy,  0.0%ni, 96.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:  12187448k total, 10686748k used,  1500700k free,    67104k buffers
    Swap:  1048568k total,   275904k used,   772664k free,  2371208k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    24252 wwwrun    20   0 2216m 161m 7484 S 20.2  1.4   0:09.25 httpd
    24750 wwwrun    20   0 1264m  51m 6196 S  9.9  0.4   0:00.93 httpd
    23996 wwwrun    20   0 2136m 279m 7292 S  6.0  2.4   0:16.76 httpd
    23326 wwwrun    20   0 2216m 276m 7376 S  2.7  2.3   0:16.81 httpd
    23582 wwwrun    20   0 2208m 293m 7132 S  2.3  2.5   0:27.30 httpd
    23688 wwwrun    20   0 1904m 208m 7304 S  1.0  1.8   0:13.62 httpd
    19507 wwwrun    20   0 2182m 229m 8096 S  0.7  1.9   0:18.65 httpd
    23616 wwwrun    20   0 1908m 239m 7092 S  0.7  2.0   0:18.48 httpd
    18304 wwwrun    20   0 2160m 334m  11m S  0.3  2.8   0:47.38 httpd
    23637 wwwrun    20   0 1830m 231m 7556 S  0.3  1.9   0:20.56 httpd
    24457 wwwrun    20   0 2196m 174m 6568 S  0.3  1.5   0:04.11 httpd
    15694 wwwrun    20   0 2199m 295m 7504 S  0.0  2.5   0:20.54 httpd
    15783 wwwrun    20   0 2172m 244m  10m S  0.0  2.1   0:17.45 httpd
    16577 wwwrun    20   0 2128m 354m 7436 S  0.0  3.0   0:28.21 httpd
    18290 wwwrun    20   0  277m 9880  964 S  0.0  0.1   0:00.05 httpd
    18379 wwwrun    20   0 2208m 211m 6864 S  0.0  1.8   0:08.57 httpd
    18480 wwwrun    20   0 2096m 305m 7540 S  0.0  2.6   0:17.72 httpd
    18791 wwwrun    20   0 1920m 251m 7244 S  0.0  2.1   0:20.24 httpd
    19348 wwwrun    20   0 2060m 310m 7388 S  0.0  2.6   0:22.07 httpd
    19619 wwwrun    20   0 2206m 235m 7340 S  0.0  2.0   0:15.30 httpd
    19999 wwwrun    20   0 2178m 144m 7132 S  0.0  1.2   0:05.50 httpd
    20697 wwwrun    20   0 2002m 193m 7276 S  0.0  1.6   0:08.12 httpd
    20838 wwwrun    20   0 1890m 127m 7260 S  0.0  1.1   0:07.44 httpd
    21407 wwwrun    20   0 1988m 290m 7708 S  0.0  2.4   0:39.33 httpd
    22252 wwwrun    20   0 2216m 295m 7732 S  0.0  2.5   0:27.04 httpd
    22723 wwwrun    20   0 2020m 187m 7444 S  0.0  1.6   0:39.80 httpd
    22753 wwwrun    20   0 1850m 206m 7448 S  0.0  1.7   0:10.62 httpd
    23174 wwwrun    20   0 1990m 229m 7332 S  0.0  1.9   0:07.71 httpd
    23383 wwwrun    20   0 1504m  76m 6372 S  0.0  0.6   0:01.25 httpd
    23720 wwwrun    20   0 1906m 225m 7080 S  0.0  1.9   0:20.12 httpd
    24778 wwwrun    20   0  970m  30m 5604 S  0.0  0.3   0:00.80 httpd
    

    Finally, mod_status output:

       Server Version: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips Domain Server PHP/5.3.24 with Suhosin-Patch mod_perl/2.0.5
              Perl/v5.10.1
    
       Server Built: May 15 2013 10:17:30
         ______________________________________________________________________________________________________________________________
    
       Current Time: Monday, 12-May-2014 16:41:08 BRT
       Restart Time: Monday, 12-May-2014 15:36:37 BRT
       Parent Server Generation: 1
       Server uptime: 1 hour 4 minutes 31 seconds
       Total accesses: 36446 - Total Traffic: 1.6 GB
       CPU Usage: u213.3 s13.65 cu0 cs0 - 5.86% CPU load
       9.42 requests/sec - 442.8 kB/second - 47.0 kB/request
       249 requests currently being processed, 0 idle workers
    
    WWRRKRKRRKRRRRWRRWWWRWCRR.......................................
    RWWWRRWRRRRKKRKRWRRRRRWWR.......................................
    WRWRCRRWRRWWRKRRWWRRWRWWW.......................................
    WRWKKWWWRWKRRWRCRRRWWRWWW.......................................
    WRWRWCRKCWWWRWWRRWWCRWRWW.......................................
    WRWKRWRRKKRKRRRRCWWKWWWRW.......................................
    RWWKWRRWKRRRWRRWWCRRRRRRW.......................................
    WWWRRWRKWWWKRWWWWRWWWKRRW.......................................
    RRRRRRRK.WKRRWRWWKRRWRRKR.......................................
    KCRRRRWRWWKRWWWRRKWRWRWRK.......................................
    

    Update 1:

    I've tried disabling mod_security2, fiddled with worker settings, but eventually Apache would consume almost 20G of RAM (I've added a 13G swapfile to prevent the server crashing).

    Finally I disable KeepAlive, and kept MaxRequestsPerChild to 100 - now Apache is using 2-3G, and the requests are being cleaned, as status output below:

       Current Time: Tuesday, 13-May-2014 17:07:48 BRT
       Restart Time: Tuesday, 13-May-2014 16:49:14 BRT
       Parent Server Generation: 0
       Server uptime: 18 minutes 33 seconds
       Total accesses: 6637 - Total Traffic: 133.2 MB
       CPU Usage: u34.43 s3.05 cu0 cs0 - 3.37% CPU load
       5.96 requests/sec - 122.5 kB/second - 20.5 kB/request
       14 requests currently being processed, 86 idle workers
    
    ..................R.............................................
    ________________RR__R_R__RC_RWC___________________..............
    R_WC__R___R_______________________________________..............
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    
  • Daniel Costa
    Daniel Costa about 10 years
    Thanks, I'll try your suggestion and take a look at KeepaliveTimeout too. I still want to know the relation between the apache "servers" (ServerLimit) and the processes shown in top -- I mean, there were 20+ apache processes in top but it was configured to spawn 10 servers max.
  • Nathan C
    Nathan C about 10 years
    I see you're using mod_perl which may be responsible. The status page shows 10 processes, so all the others are likely different parts of Apache and not handling requests directly.