apache crashing

Question

Please help with some ideas about this Apache server crash:

Here is my MPM config and errors

<IfModule prefork.c>
StartServers       5
MinSpareServers    30
MaxSpareServers   40
ServerLimit      950
MaxClients       950
MaxRequestsPerChild  2000
</IfModule>

[Sun Sep 09 03:08:03 2012] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.16 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
[Mon Sep 10 17:26:47 2012] [error] [mod_pagespeed 0.10.22.4-1633 @5088] [0910/172647:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:48 2012] [error] [mod_pagespeed 0.10.22.4-1633 @4983] [0910/172648:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:48 2012] [error] [mod_pagespeed 0.10.22.4-1633 @4915] [0910/172648:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:49 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Sep 10 17:26:49 2012] [error] [mod_pagespeed 0.10.22.4-1633 @5095] [0910/172649:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:49 2012] [error] [mod_pagespeed 0.10.22.4-1633 @5085] [0910/172649:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:49 2012] [error] [mod_pagespeed 0.10.22.4-1633 @4989] [0910/172649:ERROR:worker.cc(120)] Unable to start worker thread
[Mon Sep 10 17:26:50 2012] [alert] Child 5108 returned a Fatal error... Apache is exiting!
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Sep 10 17:26:50 2012] [emerg] (22)Invalid argument: couldn't release the accept mutex

Solution

The most relevant error seems to be:

[Mon Sep 10 17:26:49 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48

Things just go downhill from there.

This means Apache hit the limit of the maximum number of processes allowed to run.

You can view the limit by running ulimit -u. To increase the limit, run ulimit -u <number> or edit /etc/security/limits.conf (probably Red Hat specific) and add a configuration directive for Apache's user to increase the value for nproc to something more reasonable.

You can also try reducing Apache's ServerLimit and MaxClients directives.

Comments

[Michael Hampton]

It may be unlimited for root, but probably not for apache user! Be sure to check /etc/security/limits.conf or its equivalent if you aren't using a Red Hat derived distribution.

[Pi314]

ulimit -s say 10240 everything is commented out in that file

[Pi314]

ulimit -u 1024.. I am kind of learing about this limits so I don't really know what is a good value or what 1024 means (limit of opened files?)

[89c3b1b8-b1ae-11e6-b842-48d705]

Explain why you immediately believe this to be a limits issue.

[paulie4]

The article states that errno 11 from the setuid() call means the nproc limit was hit, which produces the error log message "[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48".