Apache stopped working permanently after error "Server ran out of threads to serve requests."

apache-2.2 windows-server-2008-r2 threads

11,177

what i found was this https://www.apachelounge.com/viewtopic.php?p=21108 hope it helps you

Thanks

11,177

Leonardo

Data center administrator, senior software developer, Computer Graphics researcher.

Updated on September 18, 2022

Comments

Leonardo over 1 year

I have been working with Apache for more than 8 years, but now I'm facing a severe problem and I couldn't find a solution on internet so far.

I'm running Apache 2.2.15 32bit on a Windows 2008 R2 64bit with Service Pack 1. I always used a 32bit Apache because my CGI codes (exe files) are 32bit.

The computer has a 3.10 GHz processor and 16GB RAM. The server load is always very low. The access log showed a normal request amount when the problem happened.

Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don't know how to fix this problem.

It has been many weeks I don't change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.

By using Windows Event Viewer, I have found an error log when Apache stopped working. It's the error event:

Log Name:      Application
Source:        Application Error
Date:          6/28/2016 8:08:33 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PictureToPeople
Description:
Faulting application name: httpd.exe, version: 2.2.15.0, time stamp: 0x4b8fed95
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58
Exception code: 0xc0000005
Fault offset: 0x00037188
Faulting process id: 0xbbc
Faulting application start time: 0x01d1cdfef0f8148c
Faulting application path: C:\Apache2.2\bin\httpd.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 3c1e0a43-3d85-11e6-aad3-001e8cf18221
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-06-28T23:08:33.000000000Z" />
    <EventRecordID>693</EventRecordID>
    <Channel>Application</Channel>
    <Computer>PictureToPeople</Computer>
    <Security />
  </System>
  <EventData>
    <Data>httpd.exe</Data>
    <Data>2.2.15.0</Data>
    <Data>4b8fed95</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7601.17514</Data>
    <Data>4ce7ba58</Data>
    <Data>c0000005</Data>
    <Data>00037188</Data>
    <Data>bbc</Data>
    <Data>01d1cdfef0f8148c</Data>
    <Data>C:\Apache2.2\bin\httpd.exe</Data>
    <Data>C:\Windows\SysWOW64\ntdll.dll</Data>
    <Data>3c1e0a43-3d85-11e6-aad3-001e8cf18221</Data>
  </EventData>
</Event>

It's what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It's possible to see the error message "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting". After this message, Apache never worked again. (Previous Apache error log messages were recorded very earlier and are not related.)

[Tue Jun 28 20:08:30 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 20:08:31 2016] [notice] Child 3004: Exit event signaled. Child process is ending.
[Tue Jun 28 20:08:32 2016] [info] Child 3004: Accept thread exiting.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Released the start mutex
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Terminating 99 threads that failed to exit.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: All worker threads have exited.
[Tue Jun 28 20:08:32 2016] [info] [client 1.23.96.161] (OS 10038)An operation was attempted on something that is not a socket.  : core_output_filter: writing data to the network
[Tue Jun 28 20:08:34 2016] [notice] Parent: Child process exited successfully.
[Tue Jun 28 20:08:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=284)
[Tue Jun 28 20:08:54 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:08:54 2016] [notice] Server built: Mar  4 2010 11:27:46
[Tue Jun 28 20:08:54 2016] [notice] Parent: Created child process 2468
[Tue Jun 28 20:08:54 2016] [info] Parent: Duplicating socket 308 and sending it to child process 2468
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Child process is running
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Acquired the start mutex.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting 100 worker threads.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting thread to listen on port 80.
[Tue Jun 28 20:08:56 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

After the error above, restarting Windows and Apache service didn't solve the problem (same error message). After reinstalling Apache and booting the machine, the problem was still there (same error message). So, I increased the "MaxRequestsPerChild" parameter to 300, but all worker threads were consumed in 12 seconds. So, I increased the "MaxRequestsPerChild" parameter to 500 and rebooted the machine, but all worker threads were consumed in 9 seconds as the log shows below.

[Tue Jun 28 21:55:01 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 21:55:01 2016] [notice] Child 864: Exit event signaled. Child process is ending.
[Tue Jun 28 21:55:02 2016] [info] Child 864: Accept thread exiting.
[Tue Jun 28 21:55:02 2016] [notice] Child 864: Released the start mutex
[Tue Jun 28 21:55:20 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [notice] Parent: Forcing termination of child process 368 
[Tue Jun 28 21:55:31 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=2252)
[Tue Jun 28 22:10:13 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 22:10:13 2016] [notice] Server built: Mar  4 2010 11:27:46
[Tue Jun 28 22:10:13 2016] [notice] Parent: Created child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Child process is running
[Tue Jun 28 22:10:13 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Acquired the start mutex.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting 500 worker threads.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting thread to listen on port 80.
[Tue Jun 28 22:10:22 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

Below you can see my complete original httpd.conf. It had been working correctly for months. (I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data with "xxxxx"):

ServerRoot "C:/Apache2.2"

Listen 80

LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so

<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
User daemon
Group daemon

</IfModule>
</IfModule>


ServerAdmin xxxxxxxxxxxxxxxxxxxxxxxx

  ServerName xxxxxxxxxxxxxxxxxxxxxxxx:80


DocumentRoot "xxxxxxxxxxxxxxxxxxxxxxxx"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>


<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
    Options FollowSymLinks
    Options +Includes
    Options -Indexes

    AllowOverride None

    Order allow,deny
    Allow from all

    LimitRequestBody 10485760

    SetOutputFilter DEFLATE
    SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

</Directory>

<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

<IfModule mod_headers.c>
    Header always append X-Frame-Options SAMEORIGIN
</IfModule>

<IfModule mod_expires.c>

    ExpiresActive on
    ExpiresDefault                                      "access plus 1 week"

    ExpiresByType text/css                              "access plus 1 week"

    ExpiresByType application/atom+xml                  "access plus 1 hour"
    ExpiresByType application/rdf+xml                   "access plus 1 hour"
    ExpiresByType application/rss+xml                   "access plus 1 hour"

    ExpiresByType application/json                      "access plus 0 seconds"
    ExpiresByType application/ld+json                   "access plus 0 seconds"
    ExpiresByType application/schema+json               "access plus 0 seconds"
    ExpiresByType application/vnd.geo+json              "access plus 0 seconds"
    ExpiresByType application/xml                       "access plus 0 seconds"
    ExpiresByType text/xml                              "access plus 0 seconds"

    ExpiresByType image/vnd.microsoft.icon              "access plus 1 week"
    ExpiresByType image/x-icon                          "access plus 1 week"

    ExpiresByType text/html                             "access plus 1 day"

    ExpiresByType application/javascript                "access plus 1 week"
    ExpiresByType application/x-javascript              "access plus 1 week"
    ExpiresByType text/javascript                       "access plus 1 week"

    ExpiresByType application/manifest+json             "access plus 1 year"

    ExpiresByType application/x-web-app-manifest+json   "access plus 0 seconds"
    ExpiresByType text/cache-manifest                   "access plus 0 seconds"

    ExpiresByType audio/ogg                             "access plus 1 month"
    ExpiresByType image/bmp                             "access plus 1 month"
    ExpiresByType image/gif                             "access plus 1 month"
    ExpiresByType image/jpeg                            "access plus 1 month"
    ExpiresByType image/png                             "access plus 1 month"
    ExpiresByType image/svg+xml                         "access plus 1 month"
    ExpiresByType video/mp4                             "access plus 1 month"
    ExpiresByType video/ogg                             "access plus 1 month"
    ExpiresByType video/webm                            "access plus 1 month"

  <Directory xxxxxxxxxxxxxxxxxxxxxxxx>
      ExpiresByType image/gif                             "access plus 0 seconds"
      ExpiresByType image/jpeg                            "access plus 0 seconds"
      ExpiresByType image/png                             "access plus 0 seconds"
  </Directory>



    ExpiresByType application/vnd.ms-fontobject         "access plus 1 month"
    ExpiresByType font/eot                              "access plus 1 month"

    ExpiresByType font/opentype                         "access plus 1 month"

    ExpiresByType application/x-font-ttf                "access plus 1 month"

    ExpiresByType application/font-woff                 "access plus 1 month"
    ExpiresByType application/x-font-woff               "access plus 1 month"
    ExpiresByType font/woff                             "access plus 1 month"

    ExpiresByType application/font-woff2                "access plus 1 month"

    ExpiresByType text/x-cross-domain-policy            "access plus 1 week"

</IfModule>


ErrorLog "|bin/rotatelogs logs/error.%Y-%m-%d.log 50M"

LogLevel info

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog "|bin/rotatelogs logs/access.%Y-%m-%d---%H.log 300M" common

</IfModule>

<IfModule alias_module>


    ScriptAlias /p2p/ "xxxxxxxxxxxxxxxxxxxxxxxx"

</IfModule>

<IfModule cgid_module>
</IfModule>

<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig conf/mime.types

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    AddType text/html .html
    AddHandler server-parsed .html
</IfModule>


<IfModule mpm_winnt_module>
    ThreadsPerChild      100
    MaxMemFree           100
    MaxRequestsPerChild  5000
</IfModule>


<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>


RewriteEngine On
RewriteCond %{HTTP_HOST} !^www
RewriteRule (.*) http://www.%{HTTP_HOST}$1 [L,R]

As shown above, my Apache stops responding almost instantly after launch.

My site doesn't connect to databases. It just serves dynamic pages by using CGI.

Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?

Could anyone give me any advice to solve this issue or at least to get a workaround?

Thanks in advance.

Leonardo almost 8 years

I replaced the original web server for an Apache 2.2.31 64bits compiled with Visual Studio (downloaded from Apache Haus), but the result was the same: Apache gets out of threads in a few seconds.
steve over 6 years

I have exactly the same problem. Have you find the solution ? Thanks for your response
Timoteo Ponce about 5 years

Please check stackoverflow.com/questions/6620037/…, it solved my problem

Leonardo almost 8 years

I read this before posting here. Unfortunately, it didn't help.
Leonardo about 6 years

I'm sure something like that is not my case. I even considered the case of a massive DOS attack, but the Apache logs don't show any relevant load that could justify any lack of resources. Besides, as a test, I have already configured this server's firewall to block all requests, except from my own computer, and, even this way, the Apache freezes after very few requests.