Scoreboard is full,not at MaxRequestWorkers

Question

I an using Apache and Tomcat with mod-jk in my project. My web application is in Tomcat instances. I am using Apache jmeter for testing http load on tomcat. But I'm getting the error Scoreboard is full,not at MaxRequestWorkers even with 1000 threads.

[mpm_event:error] [pid 24313:tid 3075319488] AH00485: scoreboard is full, not at MaxRequestWorkers

[mpm_event:error] [pid 24313:tid 3075319488] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

I've configured http-mpm for approximately 5000 threads.

httd-mpm.conf file is:

<IfModule mpm_event_module>
    ServerLimit              200
    StartServers             3
    MinSpareThreads          75
    MaxSpareThreads          250
    MaxClients               5000 
    ThreadsPerChild          25
    MaxRequestWorkers        5000
    MaxConnectionsPerChild   5000
</IfModule>

Can anybody please guide me what can be the problem in this configuration?

Solution

It seems that Apache tries to gracefully finish some of the processes and hangs while doing so. It might be a problem of Apache (https://bz.apache.org/bugzilla/show_bug.cgi?id=53555) or a problem of the application itself.

You can easily check it by looking at the server-status page of apache. In order to do that you need to enable the status.conf apache module. Then go to your-server.org/server-status. You will see all the processes and threads currently being used or gracefully finishing ("G"-state). If you have to many G's that don't go away, you found your problem. It would look sth like this:

Too many threads gracefully finishing...

The solution is pretty simple. Just set MaxConnectionsPerChild to 0 or comment it out (mpm_event.conf). What this does is just let the process continue it's work without restarting it. That's a good thing in our case, since restarting the process let to the problem of having to many G's filling up the scoreboard and eventually crashing Apache.

Hope that helped.

Comments

[W3Coder]

This was a livesaver - completely fixed the problem on Apache 2.4.10, which comes with Debian (Jessie) 8.9. Thanks so much!