httpd keeps crashing without any reference to why in the logs

Question

I have the logs set to debug in the hopes of tracking down what's causing the crash, but I can't find anything. Here is the error_log.

    [Thu Jan 06 10:27:35 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 19999 for (*)
[Thu Jan 06 14:47:04 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jan 06 14:47:04 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:04 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:04 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:04 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 06 14:47:04 2011] [notice] Digest: done
[Thu Jan 06 14:47:04 2011] [debug] util_ldap.c(2021): LDAP merging Shared Cache conf: shm=0xb9dc2480 rmm=0xb9dc24b0 for VHOST: server.fredfinn.com
[Thu Jan 06 14:47:04 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Thu Jan 06 14:47:04 2011] [info] LDAP: SSL support available
[Thu Jan 06 14:47:05 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Thu Jan 06 14:47:05 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Thu Jan 06 14:47:05 2011] [info] Shared memory session cache initialised
[Thu Jan 06 14:47:05 2011] [info] Init: Initializing (virtual) servers for SSL
[Thu Jan 06 14:47:05 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Thu Jan 06 14:47:05 2011] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26527 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26527 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26528 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26528 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26529 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26529 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26530 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26530 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26532 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26532 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26533 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26533 for (*)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26534 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26534 for (*)
[Thu Jan 06 14:47:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Jan 06 14:47:05 2011] [info] Server built: Aug 30 2010 12:32:08
[Thu Jan 06 14:47:05 2011] [debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem)
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 26531 for worker proxy:reverse
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[Thu Jan 06 14:47:05 2011] [debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 26531 for (*)

The logs are setup as:

ErrorLog logs/error_log

LogLevel debug

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

CustomLog logs/access_log common

CustomLog logs/access_log combined

ServerSignature On

Solution 1

Turns out mysql was eating up TONS of ram and sending me over the limit for the VPS and consequentially killing the httpd. I was running wordpress. I've removed it this morning, and the ram is back to a reasonable level. We'll see if the crashing continues. Thanks for all your help, I learned a lot!

Solution 2

Are you getting a core dump file? You can run the debugger on it to see where it's crashing. That will let you ID if the crash is in the apache core code or in one of the modules. If it's a module then you can look for bugs and updates for that module.

See this for setting where it tries to dump core:

http://httpd.apache.org/docs/2.0/mod/mpm_common.html#coredumpdirectory

Comments

[silviud]

how many virtual hosts do you have (if any) and how is the logging setup ?

[Fred]

Added log setup to bottom of post.

[PP.]

I had an issue that may be similar - question: are you running your httpd binary from the location it was compiled to be run from?

[Fred]

Ok cool! I set that in the httpd to the log dir. Hopefully that will give me something to work with... Thanks!

[Derek Downey]

"You can run the debugger on it to see where it's crashing" which debugger are you referring to?

[user2987902]

I think he means gdb.

[JOTN]

Yes, gdb, dbx, or whatever is on you system: gdb /path/to/httpd /path/to/core

[Fred]

Goodbye wordpress. That solved it.