apache out of memory

Question

I'm using CentOS 5 on a VPS with 512MB memory. I find httpd often runs out of memory, which lags the whole system. But I cannot find out the reason.

When I use top to monitor processes, sometimes one of the httpd takes out 90+% memory, then SSH hangs up for seconds. When top refreshes again, that httpd has been killed by kernel. The issue repeats a few minutes later, again and again.

Here is the system log: http://pastebin.com/bszy9ahq

My Apache prefork conf:

<IfModule prefork.c>
StartServers 1
MinSpareServers 3
MaxSpareServers 7
ServerLimit      128
MaxClients       128
MaxRequestsPerChild 4000
</IfModule>

Here is httpd -V:

Server version: Apache/2.2.3
Server built:   Mar 24 2012 13:32:42
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Is there any way to know what script cause the memory issue, or just apache related? And how to resolve...

Solution

The system has run out of memory and has decided to kill apache to make room. It's not necessarily apache which is using all the memory. 512Mb might be tight for 128 httpd processes - but that figure can vary dramatically. Have you looked to see how much memory an httpd instance uses? Don't bother trying to work it out from a static snapshot - plot the memory usage (less buffers and cache) against the number of httpd instances and do a least squares regression - the slope is the amount of memory used per instance. Then find out how much memory the system uses when Apache is switched off. Depending on the amount of content being read from disk, you should aim for a minimum of ~20% memory free for cache/buffers.

Have you tuned your httpd.conf? Removing all redundant modules? Keepalives should be enabled but the timeout should be 2 seconds at most. Enable output compression and caching for static content.

If you suspect a memory leak in Apache then try reducing MaxRequestsPerChild.

sometimes one of the httpd takes out 90+% memory

This rather suggests a runaway script / module.

any way to know what script

Not from the information you've provided - there's nothing to tell us what scripting language you are using nor how it is invoked from the httpd. If it's PHP, then try adding an autoprepend script like....

<?php

register_shutdown_function('logmem');

function logmem()
{
    apache_note('x-php-peak-mem', memory_get_peak_usage());
}

and report the value in your access log:

"%{x-php-peak-mem}n"

Obviously this is only going to report requests which complete successfully - but it's a start.

Again, if it's PHP, then setting the memory limit too high (although this also applies to other OO languages) may be counter productive by suppressing garbage collection.

(BTW if this is running out of memory after a few minutes then "StartServers 1" is just plain silly)

Comments

[Clippit]

I migrated from another 512M VPS some weeks ago. However almost the same sites and scripts worked fine in the past.

[Tom O'Connor]

Are the servers identical? If i ran a directory comparison diff against them, would I find no differences?

[Clippit]

I reinstalled everything on the new server and restored site files, so... you know

[Tom O'Connor]

The answer is "no, then.