"No space left on device: AH00023: Couldn't create the mpm-accept mutex" when restarting httpd

Question

My server sometimes gets filled up with connections in apache stuck in a "Sending Reply" state, requiring me to restart apache. Most of the time this works, but sometimes I will get this error when trying to restart apache instead:

Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe"
for details.

Running systemctl status httpd.service or journalctl -xe as suggested then brings back the following relevant information:

Nov 15 06:24:06 hostname.biologyreporter.com systemd-logind[874]:
 Failed to remove runtime directory /run/user/1067: Device or resource busy

Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 [Fri Nov 15 06:24:27.255594 2019] [core:emerg] [pid 29509:tid 47498001208384]
 (28) No space left on device: AH00023: Couldn't create the mpm-accept mutex  
 
Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 (28) No space left on device: could not create accept mutex  
 
Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]:
 AH00015: Unable to open logs

Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]:
 httpd.service: control process exited, code=exited status=1  
 
Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]:
 Failed to start Apache web server managed by cPanel EasyApache.
 -- Subject: Unit httpd.service has failed

It may then take ~10 minutes or so for apache to actually restart and the website to be functional again. I don't know why it is saying "No space left on device" as running df, for example, returns the following (nowhere is 100% of space taken):

Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs         3973024         0   3973024   0% /dev
tmpfs            3983400         0   3983400   0% /dev/shm
tmpfs            3983400    255296   3728104   7% /run
tmpfs            3983400         0   3983400   0% /sys/fs/cgroup
/dev/sda2      952008348 137586024 766039760  16% /
/dev/sda1         999320    134892    795616  15% /boot
/dev/loop0       3997376      8856   3778808   1% /tmp
tmpfs             796684         0    796684   0% /run/user/0
tmpfs             796684         0    796684   0% /run/user/1022

I've been unable to find any reliable information on the cause and solution of the specific error(s) as listed above

• No space left on device: AH00023: Couldn't create the mpm-accept mutex and
• Failed to remove runtime directory /run/user/1067: Device or resource busy

What should I do to solve this so that these errors don't show up and apache always restarts without issues?

Solution 1

This is most likely caused by the semaphore limit of the OS, and Apache not properly cleaning up after itself.

See more details about semaphore limits https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-setting_semaphores-setting_semaphore_parameters

Those errors mean that there is a lack of inner-process communication resources in the system, such as semaphores or shared memory segments. Try running next to see if the semaphore limit is reached and see results:

ipcs -s | wc -l

cat /proc/sys/kernel/msgmni

cat /proc/sys/kernel/sem

Execute the following command via SSH or console and see if it helps :

ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}'

ipcs -s lists currently used semaphores.

awk -v user=apache filters out the ones, owned by apache user and next part executes icprm -s "id" for each of it which removes that semaphore.

I.e. whole command removes all semaphores owned by apache user

Solution 2

While the answer by @dzup4uk seems to be able to point out what the issue is, in order to actually solve the issue you have to ensure that the semaphores get cleaned up. ipcs command will show what the issue is but in order to actually solve the issue you have to do some cleaning which can be done with

ipcrm -a

Comments

[Orphans]

what if you do "df -i" instead?

[user548654]

Loads of free space for everything

[user548654]

what does the command ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}' do?

[NStorm]

@user548654, ipcs -s lists currently used semaphores. awk -v user=apache filters out the ones, owned by apache user and next part executes icprm -s "id" for each of it which removes that semaphore. I.e. whole command removes all semaphores owned by apache user.

[dzup4uk]

@NStorm thank you so much for the explanation!

[bhu Boue vidya]

OMG THANK YOU! ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}' command fixed it for me!!!

[JimDeadlock]

Beautiful answer, helped me out thanks

[codenaugh]

is there a fix for apache not cleaning up after itself?

[JeremyCanfield]

Good tip here @Barnabas Busa. One small observation. On my system, I found semaphores being used by root associated with critical system processes. Perhaps it would be best to instead use ipcrm -s <semid> so that a semaphore associated with a critical system process is not removed?