What is the meaning of "AH00485: scoreboard is full, not at MaxRequestWorkers"?

Question

My Environment

• CentOS 6.4 X86_64
• Apache 2.4.4
• PHP 5.4.16 (FPM)
• 2 Intel Xeon E5-2620 @ 2.00GHz (8 core, 16 threads in each processor)
• 48GB RAM registered memory.
• 3 Hard Disk 15RPM 145GB in RAID0 (by BIO

Interesting Variables

    <IfModule mpm_event_module>
        StartServers             2
        ThreadLimit             196
        MinSpareThreads         96
        MaxSpareThreads        192
        ThreadsPerChild         96
        MaxRequestWorkers      192
        MaxConnectionsPerChild   96
    </IfModule>

Apache Server Status

Server Version: Apache/2.2.4 (Unix) OpenSSL/1.0.1e mod_fastcgi/mod-fastcgi-SNAP-0910052141
Server Built: May 24 2013 16:48:07

---

Current Time: Monday, 17-Jun-2013 09:48:11 COT
Restart Time: Monday, 17-Jun-2013 08:35:14 COT
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 hour 12 minutes 57 seconds
Server load: 0.05 0.10 0.09
Total accesses: 14144 - Total Traffic: 349.7 MB
CPU Usage: u.28 s.25 cu0 cs0 - .0121% CPU load
3.23 requests/sec - 81.8 kB/second - 25.3 kB/request
1 requests currently being processed, 191 idle workers

  PID | Connections       | Threads     | Async connections
      | total | accepting | busy | idle | keep-alive | closing
  ==============================================================
18997 | 3     | yes       | 1    | 95   | 0          | 3
18485 | 0     | yes       | 0    | 96   | 0          | 0
  ==============================================================
Sum   | 3     |           | 1    | 191  | 0          | 3

Error Log

The error message is

[Mon Jun 17 09:32:45.680842 2013] [mpm_event:error] [pid 8574:tid 140185091581760] AH00485: scoreboard is full, not at MaxRequestWorkers

This appears every few seconds. I don’t understand it. How can I fix it?

Solution 1

We had the same problem on Apache 2.4.6. After monitoring the server and adjusting the setting for several hours it appears to us that Apache may have a bug. What appears to happen is that the server processes occasionally goes into the G state (Gracefully finishing) and restarts to accept new requests, that's normal. What is not normal is that for some reason this can take up to a few minutes to restart. If you only have a few server process running and they all go into the G state at the same time then your scoreboard fills up and you won't be able to server any more requests.

What we did was increase the number of servers so there is a less of a chance that they will all go into the G state at the same time. Also make sure you allocate at least 25 threads (MaxRequestWorkers) for each server process because that appears to be the default (i.e. if 5 Servers x 25 ThreadsPerChild = 125 MaxRequestWorkers). You can change ThreadsPerChild if you like, we left it at default. If you don't allocate enough threads the additional servers will not start. We left MinSpareThreads at the default value which is 25 and the default for MaxSpareThreads which is 75. If you do modify these settings, the value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild. Also MaxRequestWorkers must be equal to or less than the ServerLimit.

Here is what worked for us but it might not be the best configuration for you.

StartServers 3
MinSpareServers 5
MaxSpareServers 10
ServerLimit 250
MaxRequestWorkers 250
MaxConnectionsPerChild 1000
KeepAlive Off

Edit: This is a confirmed bug in httpd's mpm_event module which might not be fixable through configuration.
The linked bugtracker entry has a presumed patch and more discussion about how to fix this until a new version of the event module is officially released.

Solution 2

Seeing the same issue.

Apache 2.4.7-1ubuntu4.4 on Ubuntu 14.04
Server Version: Apache/2.4.7 (Ubuntu)
Server MPM: event
Server Built: Mar 10 2015 13:05:59 

We particularly can cause this behaviour by reloading apache.

What we then see, are a couple of old processes that don't stop:

root     28192  0.0  0.8 103772  8648 ?        Ss   Mar16   0:03 /usr/sbin/apache2 -k start
www-data  2530  0.3  2.1 865188 21516 ?        Sl   06:26   0:54  \_ /usr/sbin/apache2 -k start
www-data  2531  0.2  2.1 865436 21892 ?        Sl   06:26   0:51  \_ /usr/sbin/apache2 -k start
www-data  3299  0.3  2.0 864140 20628 ?        Sl   06:46   0:51  \_ /usr/sbin/apache2 -k start
www-data  7305  0.3  2.1 865100 21504 ?        Sl   08:36   0:37  \_ /usr/sbin/apache2 -k start
www-data 11952  0.2  1.8 863004 19268 ?        Sl   10:46   0:06  \_ /usr/sbin/apache2 -k start
www-data 13284  0.0  0.6 103772  6692 ?        S    11:18   0:00  \_ /usr/sbin/apache2 -k start
www-data 13553  2.1  2.0 866156 21248 ?        Sl   11:23   0:01  \_ /usr/sbin/apache2 -k start

Notice the 'older' and 'newer' PID's and start times. ^^

PID Connections     Threads Async connections
total   accepting   busy    idle    writing keep-alive  closing
7305    14  no  0   0   0   0   0
2530    13  no  0   0   0   0   0
3299    7   no  0   0   0   0   0
13553   65  no  17  8   0   25  25
2531    15  no  0   0   0   0   0
11952   10  no  0   0   0   0   0
Sum 124     17  8   0   25  25

GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGW_WWWW__W_W_W_WWWWWWW__WWGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGG

Comments

[rustyx]

Your MaxConnectionsPerChild setting is way too low for production use. Furthermore, setting it to anything other than 0 is only meant to be done on Windows because it leaks memory internally.

[dhaupin]

Apache error_log also gives hints: MaxRequestWorkers of 40 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 25

[Hamish Moffatt]

MaxSpareServers/MinSpareServers aren't applicable to mpm_event. I'm not sure what you meant here because the numbers are way too low to be MaxSpareThreads/MinSpareThreads.

[Yves Martin]

Also faced this trouble on Debian at Apache2 log rotation. Refer to support.plesk.com/hc/en-us/articles/…

[Jérôme]

The patch mentionned in this answer was merged in 2.4.25. I'm here because I have the problem, although I'm using 2.4.25. Apparently, it appeared on a reload triggered by logrotate and the processes continue to write in error.log.1. error.log only mentions the reload.

[Hayden Thring]

not an answer but comment

[mailq]

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review