Hung Processes in Apache?

Question

I'm running Apache 2.2.21 with the MPM prefork on a dedicated server. More details:

Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/1.0.0-fips DAV/2 SVN/1.7.0 mod_python/3.3.1 Python/2.6.5 mod_bwlimited/1.4 PHP/5.3.6

Traffic is usually somewhere between 10-30 requests/sec, there's 12GB of RAM, and we've tuned MaxClients fairly conservatively (250). We do see spikes of usage for various reasons (on an older server, we hit our MaxClients of 100 several times during these spikes).

Anyways, this is a new server. After running for a while, our Apache status starts to look something like this:

GGG_._._RC_.G..C.G_G.C_G..C_.CG_._._G__W____..R.WCR_.W..G_......

The G's ("Gracefully finishing") are getting stuck. They only disappear if I restart Apache. This will definitely cause us to hit our MaxClients limit without regular monitoring/fixing. I read around on the internet and there seems to be a bug in Apache that is similar to this, but happens under different conditions. It was also apparently fixed in version 2.2.14.

I'm included the stacktrace of a hung process for your inspection.

#0  0x000000350c6f119e in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x000000350c67c138 in _L_lock_9164 () from /lib64/libc.so.6
#2  0x000000350c679a32 in malloc () from /lib64/libc.so.6
#3  0x000000350c66fcfb in __libc_message () from /lib64/libc.so.6
#4  0x000000350c675676 in malloc_printerr () from /lib64/libc.so.6
#5  0x000000350c675aa1 in malloc_consolidate () from /lib64/libc.so.6
#6  0x000000350c677f38 in _int_free () from /lib64/libc.so.6
#7  0x0000003906c64cbb in my_once_free () at my_once.c:117
#8  0x0000003906c5d6ff in my_end (infoflag=0) at my_init.c:170
#9  0x0000003906c5c547 in mysql_server_end () at libmysql.c:209
#10 0x00007f34ac195be8 in zm_shutdown_mysqli (type=<value optimized out>, module_number=22)
    at /home/cpeasyapache/src/php-5.3.6/ext/mysqli/mysqli.c:834
#11 0x00007f34ac2b825f in module_destructor (module=0x1eafce0) at /home/cpeasyapache/src/php-5.3.6/Zend/zend_API.c:2098
#12 0x00007f34ac2be945 in zend_hash_apply_deleter (ht=0x7f34ac988aa0, p=0x1eafc80) at /home/cpeasyapache/src/php-5.3.6/Zend/zend_hash.c:614
#13 0x00007f34ac2bebd8 in zend_hash_graceful_reverse_destroy (ht=0x7f34ac988aa0) at /home/cpeasyapache/src/php-5.3.6/Zend/zend_hash.c:649
#14 0x00007f34ac2b3085 in zend_shutdown () at /home/cpeasyapache/src/php-5.3.6/Zend/zend.c:759
#15 0x00007f34ac26017a in php_module_shutdown () at /home/cpeasyapache/src/php-5.3.6/main/main.c:2146
#16 0x00007f34ac260229 in php_module_shutdown_wrapper (sapi_globals=<value optimized out>)
    at /home/cpeasyapache/src/php-5.3.6/main/main.c:2118
#17 0x00007f34ac33a461 in php_apache_child_shutdown (tmp=<value optimized out>)
    at /home/cpeasyapache/src/php-5.3.6/sapi/apache2handler/sapi_apache2.c:399
#18 0x00007f34ae59dea4 in run_cleanups () from /usr/local/apache/lib/libapr-1.so.0
#19 0x00007f34ae59cd72 in apr_pool_destroy () from /usr/local/apache/lib/libapr-1.so.0
#20 0x00000000004cc004 in clean_child_exit ()
#21 0x00000000004ccd00 in child_main ()
#22 0x00000000004cce62 in make_child ()
#23 0x00000000004cd107 in perform_idle_server_maintenance ()
#24 0x00000000004cd664 in ap_mpm_run ()
#25 0x000000000042e24f in main ()

It looks like the problem happens when our PHP script closes the mysqli connection. There's a hang when attempting to free memory. Anyone with a similar configuration - Apache 2.2.21, PHP 5.3.6, MySQL/mysqli (5.1.56) - having similar issues?

Anyone know what I can attempt to do to fix this? Upgrade MySQL/Apache/PHP? I'm happy to provide more information if it helps.

Thanks!

UPDATE: Seems like MySQL isn't really involved in the problem. Here's another stacktrace of a hanging process that only includes PHP:

#0  0x000000350c6f119e in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x000000350c67c138 in _L_lock_9164 () from /lib64/libc.so.6
#2  0x000000350c679a32 in malloc () from /lib64/libc.so.6
#3  0x000000350c66fcfb in __libc_message () from /lib64/libc.so.6
#4  0x000000350c675676 in malloc_printerr () from /lib64/libc.so.6
#5  0x000000350c675aa1 in malloc_consolidate () from /lib64/libc.so.6
#6  0x000000350c677f38 in _int_free () from /lib64/libc.so.6
#7  0x00007f532accb951 in zend_mm_shutdown (heap=0x2327aa0, full_shutdown=1, silent=<value optimized out>) at /home/cpeasyapache/src/php-5.3.6/Zend/zend_alloc.c:1648
#8  0x00007f532ac951af in php_module_shutdown () at /home/cpeasyapache/src/php-5.3.6/main/main.c:2159
#9  0x00007f532ac95229 in php_module_shutdown_wrapper (sapi_globals=<value optimized out>) at /home/cpeasyapache/src/php-5.3.6/main/main.c:2118
#10 0x00007f532ad6f461 in php_apache_child_shutdown (tmp=<value optimized out>) at /home/cpeasyapache/src/php-5.3.6/sapi/apache2handler/sapi_apache2.c:399
#11 0x00007f532cfd2ea4 in run_cleanups () from /usr/local/apache/lib/libapr-1.so.0
#12 0x00007f532cfd1d72 in apr_pool_destroy () from /usr/local/apache/lib/libapr-1.so.0
#13 0x00000000004cc004 in clean_child_exit ()
#14 0x00000000004ccd00 in child_main ()
#15 0x00000000004cce62 in make_child ()
#16 0x00000000004cd107 in perform_idle_server_maintenance ()
#17 0x00000000004cd664 in ap_mpm_run ()
#18 0x000000000042e24f in main ()

UPDATE 2: Turns out this is a known problem for some systems.

http://docs.cpanel.net/twiki/bin/view/EasyApache3/EA3KnownIssues#Bug:%20Apache%202.2%20Child%20Processes

I've noticed I'm also getting "C" status Apache processes hanging around for quite some time (>3000 seconds). I've written a cron to kill processes hanging in either "G" or "C" status for a long time... but that's a band-aid. I'd like to fix the problem.

One thing I noticed, and also changed, about my configuration was that I wasn't setting thie "GracefulShutdownTimeout" directive:

http://httpd.apache.org/docs/2.2/mod/mpm_common.html#gracefulshutdowntimeout

I added this and changed it to 30 seconds. We'll see if that also helps.

More Info: Here's the output of cat /proc/PROCESS_ID/status for one of the "G" processes:

Name:   httpd
State:  S (sleeping)
Tgid:   14867
Pid:    14867
PPid:   30017
TracerPid:      0
Uid:    99      99      99      99
Gid:    99      99      99      99
Utrace: 0
FDSize: 64
Groups: 99 
VmPeak:   355752 kB
VmSize:   222996 kB
VmLck:         0 kB
VmHWM:    191120 kB
VmRSS:     77928 kB
VmData:    62300 kB
VmStk:        96 kB
VmExe:      1032 kB
VmLib:     24736 kB
VmPTE:       488 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/95107
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 000000018c0046eb
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   ffffff
Cpus_allowed_list:      0-23
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list:      0-1
voluntary_ctxt_switches:        24708
nonvoluntary_ctxt_switches:     2651

Loaded Modules in Apache:

 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 log_config_module (static)
 logio_module (static)
 mime_magic_module (static)
 expires_module (static)
 setenvif_module (static)
 ssl_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 dav_module (static)
 status_module (static)
 autoindex_module (static)
 info_module (static)
 suexec_module (static)
 cgi_module (static)
 dav_fs_module (static)
 dav_lock_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 userdir_module (static)
 alias_module (static)
 rewrite_module (static)
 so_module (static)
 python_module (shared)
 dav_svn_module (shared)
 authz_svn_module (shared)
 bwlimited_module (shared)
 php5_module (shared)

Loaded Modules in PHP:

[PHP Modules]
bcmath
Core
ctype
curl
date
dom
eAccelerator
ereg
exif
filter
gd
gettext
hash
iconv
imap
json
libxml
mbstring
mcrypt
memcache
mysql
mysqli
openssl
pcre
PDO
pdo_mysql
pdo_sqlite
posix
Reflection
session
SimpleXML
sockets
SPL
SQLite
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
zlib

[Zend Modules]
eAccelerator

Solution 1

Try setting Keepalive Off. I tend to do this automatically, especially on IO-contended systems, like virtual machines.

Solution 2

Some general guidelines I work by, sorry if this is not immediately feasible for you:

• use the worker MPM
• avoid mod_<language>
• instead use the fcgi interfaces PHP, Python etc. provide
• use mod_fcgid

If you're not modifying HTTP headers or similar in your application, you might be able to make the switch to fcgi without any code changes.

What you gain by this is a much cleaner separation of the various elements that take play in this setup. The httpd will be a lot more robust against any bugs (performance-, but also security-wise) that third party modules inflict.

Comments

[Tom O'Connor]

What's your Keepalive setting like?

[thexfactor]

Timeout 300 TraceEnable On ServerSignature Off ServerTokens Full FileETag All StartServers 40 <IfModule prefork.c> MinSpareServers 10 MaxSpareServers 20 </IfModule> ServerLimit 250 MaxClients 250 MaxRequestsPerChild 1000 KeepAlive On KeepAliveTimeout 2 MaxKeepAliveRequests 30

[thexfactor]

I'm going to test this out now, and see if it makes a difference. I have a feeling it's a problem with the configuration of Apache + modules installed there and PHP + modules installed there.

[Mircea Vutcovici]

Can you please add to your question the output of cat /proc/<hanged_appache_PID>/status?

[Janne Pikkarainen]

@thexfactor: Can you please add to your question all the Apache modules you have in use? Modules such as mod_security or mod_proxy can sometimes add bugs similar to yours.

[Hayden Thring]

@thexfactor where did you get with this ?

[thexfactor]

Because we're using PHP and Apache 2.2.x, we need to use the prefork MPM: php.net/manual/en/install.unix.apache2.php Do you think there is no real difference in terms of performance between Apache 1.x and 2.x or 2.0.x and 2.2.x? Or that we would be able to use Apache 2.x with PHP as long as we use the FastCGI Process Manaager (FPM)?

[al.]

There's huge difference in terms of performance between worker and prefork MPM. Also, yes, you should be able to use PHP in a FastCGI setup. FPM would probably help, but it can also work stable with the original FastCGI interface. This somewhat depends on the modules and functions you're using, so anyway you need to do some testing before putting this into action.