Apache cannot fork: cannot allocate memory

Question

I am running apache on Fedora with mod_perl. The server forks a process to send an email frequently - and the fork exits.

Suddenly, I got: Cannot fork: Cannot allocate memory. There was plenty of free memory (I did not look too closely, but there was more memory free than in use), not too many apache processes running - 10?, low cpu usage, close to 0. Restarting the server httpd graceful resolved the issue.

What would cause something like this? Is it possible that I got into some kind of a state where the problem was allocating too many processes in the system - and with dropping load failed to recover?

Unfortunately I do not have too much info as I was panicking and trying to get the site up instead of running diagnostics. :-) So my question is very general: is there anything that's not obvious that would create this behavior when too many processes are allocated? Anything specific to forking from within apache? etc

Thank you!

Solution

It sounds like you are hitting resource limits. Look at pam_limits.so and /etc/security/limits.conf. These allow an administrator to set explicit limits on the resources a user can use so that one user can't hog all the system resources. If Apache is hitting a limited specified by this, it will return an error like above even though there are plenty of resources other users can use. There is also the limit or ulimit command in most shells, but I don't think that's the problem.

Comments

[Coren]

can you be more precise about this ? how much is "plenty of free memory" ? how much is "not too many processes running" ? Did you run an htop-like tool to see what consumes most of the memory ?

[Nikhil Mulley]

Yes, can you be also generous in sharing your apache config (mostly performance tuning specific)

[phemmer]

can you provide the output of sysctl -a 2>/dev/null | grep '^vm' and cat /proc/meminfo. Preferably while the issue is occurring. If this is not possible, still post, there may be something blatantly obvious.

[Smiley]

Are you running this on any virtual machine? I have a similar problem, although with another software.