Apache crash : Cannot allocate memory

Question

I am experiencing a recurring issue why my ubuntu server complete becomes unavailable around the same time for the past two days. I cant even SSH into it during the crash.

The Ubuntu server is on a amazon aws t2.micro instance, running on 1 gb of ram , Ubuntu 16.04 The apache error logs are shown below

[Wed Aug 30 18:02:23.710072 2017] [autoindex:error] [pid 7505] [client 60.191.38.77:57957] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi,$
[Wed Aug 30 18:02:23.710126 2017] [:error] [pid 7505] [client 60.xx.xx.xx:57957] script '/var/www/html/404.php' not found or unable to stat
[Wed Aug 30 19:11:54.375001 2017] [autoindex:error] [pid 11307] [client 45.55.21.189:52050] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi$
[Wed Aug 30 19:11:54.376134 2017] [:error] [pid 11307] [client 45.xx.xx.xx:52050] script '/var/www/html/404.php' not found or unable to stat

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 20:50:00.570286 2017] [core:notice] [pid 31139] AH00051: child pid 17670 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:03:27.234926 2017] [core:notice] [pid 31139] AH00051: child pid 18507 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354905 2017] [core:notice] [pid 31139] AH00051: child pid 18511 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:27.354927 2017] [core:notice] [pid 31139] AH00051: child pid 18512 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:42.865027 2017] [core:notice] [pid 31139] AH00051: child pid 18506 exit signal Aborted (6), possible coredump in /etc/apache2
[Wed Aug 30 21:03:46.984235 2017] [core:notice] [pid 31139] AH00051: child pid 18529 exit signal Aborted (6), possible coredump in /etc/apache2
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[crit] Memory allocation failed, aborting process.
[Wed Aug 30 21:14:50.194072 2017] [core:notice] [pid 31139] AH00051: child pid 18605 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Aug 30 21:14:50.482541 2017] [core:notice] [pid 31139] AH00051: child pid 18618 exit signal Aborted (6), possible coredump in /etc/apache2
[Thu Aug 31 07:09:50.271441 2017] [mpm_prefork:notice] [pid 1321] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Thu Aug 31 07:09:50.274044 2017] [core:notice] [pid 1321] AH00094: Command line: '/usr/sbin/apache2'

The server becomes inaccessible around this time Wed Aug 30 19:11:54 based on the logs and traffic monitor on AWS.

I have little understanding of how Apache works in terms of memory and concurrent connections. The website I have in it has low traffic. Is there some config file I might have messed up? .BTW memory_limit is -1 no limit as per the PHP docs Or what can I do to find the cause of the problem, I am thinking of migrating the server files to a AWS beanstalk instance (A deploying aide plus scaling aide,AFAIK) a fresh start.

Is there away I could know the script(s) that might have caused the crash.I have too many scripts plus running on laravel PHP framework + third party scripts would really make it hard to outline which one is the suspect.

I would greatly appreciate an answer that explains the error log.i.e

• A script did this...

• Apache seems to have done this...

• Ubuntu then did this...

• Which led to this...

Solution

Borrowing from what @arkascha said in the comments the crash was caused by some script that was using a lot of memory and not well optimised. The following steps will help others facing similar problems.

1. Limit the php script memory limit to something reasonable according to what you running.
2. Have Swap files, really important I didn't have one which caused the whole system to crash during memory intensive processing. (In my case Apache was using all of the available memory from my 1GB, literally causing the system to crash) (A swap file is basically a file on a hard disk used to provide space for programs that have been transferred from the processor's memory. An extra RAM,but slow)
3. Find the root of the problem (I haven't found a technique or tool outline php memory usage but i think there some available). Another way is too look for scripts with long queries (multiple joins,etc) these are typically the cause.
4. Probably obvious but maybe not so obvious, get some additional RAM for your server. You webapp may require more than you currently have.
5. Caching data. There are many techniques in doing this. Basically it involves loading static data where dynamic loading is not really needed.

Comments

[arkascha]

"memory_limit is -1"... why on earth did you do that? This setting is there specifically to prevent such issues if you have poorly programmed scripts.

[arkascha]

That limit "kills" a script that has gone wild. That protects the http server itself to get unresponsive or even crash the system if you push a foul script into it. The limit to set depends on the purpose, what the scripts are supposed to do, obviously. For text (markup) generation and form processing the default values are just fine and small enough. For image manipulation you might need more sure. But typically such issues arise from scripts trying to load copies and copies of long lists into memory to process them instead of following a sequential processing strategy which scales just fine.

[arkascha]

Sure, there is some issue with your scripts. Typically such problems arise when people query a database and try to fetch all result entries at once instead of processing them sequentially. Or something similar. So copies of copies of elements of a long list get instantiated in memory in parallel instead of one after another, releasing the memory inbetween each element again.