Why did my apache2 crash? Althoought getting much more space and memory on my server

Question

My website stopped loading on the browser, and worked back again just after I restarted the apache2.

I use Amazon AWS ububuntu server moved from t2.micro to t2.large (vCPU: 2, CPU Credits / hour: 36, Mem (GiB) 8) 3 months ago.

This is the log of the apache2:

[Tue Dec 15 09:04:41.976924 2015] [autoindex:error] [pid 31049] [client 66.66.66.666:53331] AH01276: Cannot serve directory /var/www/html/wp-content/themes/twentyfifteen-child/images/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive, referer: http://www.example.com/blog/
[Tue Dec 15 18:28:59.565037 2015] [autoindex:error] [pid 31987] [client 50.115.68.210:13697] AH01276: Cannot serve directory /var/www/html/wp-content/themes/twentyfifteen-child/images/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive, referer: http://www.example.com/wp-content/themes/twentyfifteen-child/style.css?ver=4.3.1
[Wed Dec 16 02:34:52.643069 2015] [mpm_prefork:error] [pid 27704] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed Dec 16 10:40:04.422974 2015] [core:notice] [pid 27704] AH00051: child pid 28345 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423039 2015] [core:notice] [pid 27704] AH00051: child pid 28512 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423089 2015] [core:notice] [pid 27704] AH00051: child pid 28347 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423174 2015] [core:notice] [pid 27704] AH00051: child pid 27850 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423220 2015] [core:notice] [pid 27704] AH00051: child pid 13545 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423259 2015] [core:notice] [pid 27704] AH00051: child pid 31810 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423297 2015] [core:notice] [pid 27704] AH00051: child pid 28520 exit signal Segmentation fault (11), possible coredump in /etc/apache2

The problem really started around 10:40 (you can see on the log)

And this is the MaxRequestWorkers.

<IfModule mpm_prefork_module>
        StartServers              5
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers        256
        MaxConnectionsPerChild   1000
</IfModule>

Can anyone help me debug the problem on my server?

what might be the problem?

EDIT:

GBD Output:

(gdb) gdb apache2 -core /tmp/apache-coredumps/core
[1]+  Stopped                 gdb /usr/sbin/apache2
root@ip-66-66-66-66:/# db apache2 -core /tmp/apache-coredumps/core
db: command not found
root@ip-66-66-66-66:/# gdb apache2 -core /tmp/apache-coredumps/core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from apache2...Reading symbols from /usr/lib/debug//usr/sbin/apache2...done.
done.
[New LWP 6194]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  _zend_mm_free_int (heap=0x7f4b910ecfb0, p=0x7f4b910f0e70) at /build/php5-pO28mL/php5-5.5.9+dfsg/Zend/zend_alloc.c:2104
2104    /build/php5-pO28mL/php5-5.5.9+dfsg/Zend/zend_alloc.c: No such file or directory.

Solution

Your server might just had a simple overload. I can't tell, even for a t2.micro, because it depends on too much parameters.

There are options to prevent crashes like these. One of the simplest is to install Monit. This is a service that can monitor Apache, and then restart it if it sees that there are too many connections. You can use Monit for other processes as well.

Comments

[serverliving.com]

Do you see huge number of requests in access logs?

[Froggiz]

[client 66.66.66.666:53331] there is something wrong here !

[Magellan]

@Hatzil666 Go do a proper sizing exercise on your application. You're using prefork, so you need to make sure that your application memory per process multiplied by your MaxRequestWorkers is roughly 75% of total system memory.

[Admin]

@Magellan the servers ram is almost 8000 Mem: 7983 1508 6475 28 163 992. i changed it to 1000 just for a test and i get this error in this dir: /etc/apache2/mods-enabled/mpm_prefork.conf after restarting apache2: AH00180: WARNING: MaxRequestWorkers of 1000 exceeds ServerLimit value of 256 servers, decreasing MaxRequestWorkers to 256. To increase, please see the ServerLimit directive. any idea why?

[Admin]

I attached the httpd and this happens once in a long time. when I try to exit gdb it asks if i want to cancle the debugging session. is that ok ? Or there is a way to keep it going while closing it.

[Admin]

Can it give me the exact error of what's the crashes?

[Admin]

Hey, I looked into Google Analytics and noticed there was an extreme increase in the traffic than usual (18 sessions, usually i have 2-10) at the time it crashed, and I really doubt the t2.large server can be the problem. Can it be because the MaxRequestWorkers 256 is set to 256? (these are the old settings)?

[Admin]

Hey, I looked into Google Analytics and noticed there was an extreme increase in the traffic than usual (18 sessions, usually i have 2-10) at the time it crashed, and I really doubt the t2.large server can be the problem. Can it be because the MaxRequestWorkers 256 is set to 256? (these are the old settings)?

[Admin]

I added a GBD output

[Admin]

Done! Edited my post. Please help me solve this :'( it's like this for a long time so frustrating

[Rick Sanchez]

askubuntu.com/questions/753659/…