Running any command returns "Cannot allocate memory" on Ubuntu Server

Question

I’m using Ubuntu 14.04. Recently, when I login via SSH with my user with sudo privileges, every command I run results in a “Cannot allocate memory” error. Here are a few I tried at my console

myuser@mymachine:~$ whoami
-bash: fork: Cannot allocate memory
myuser@mymachine:~$ uname -a
-bash: fork: Cannot allocate memory

Even if I try sudo reboot now I get the above error, so I don’t know what else I can try to unlock my instance. The host is DigitalOcean if that matters.

Edit: Per the answer/suggestion given here is the output of "free"

myuser@mymachine:~$ free
-bash: fork: Cannot allocate memory

Solution 1

Solution

As it says in the error messages, your machine has run out of memory. This can be for a number of reasons, but basically, something is eating up all of your memory and not leaving any left for even basic command usage.

I would suggest that you reboot your droplet (just go to your client control panel and select "Reboot"), ssh in and then run top or htop. Keep an eye on the memory usage and see what process is using up all the memory. From there, try either

1. Killing/Removing the faulty program/process
   
   WARNING: PLEASE do your research on if the process is an essential system process, first! If a system process is causing memory issues, don't just kill it, do research on it and for specific ways to deal with it.
2. Changing configuration for that program/process so that it doesn't eat up all of your memory.

---

Suggestions for preventing the issue from happening again

• Something good to do is to add swap memory, as it allocates more memory if you're running out.
• Whenever you install programs, make sure you configure them correctly so that they don't perform in unintended ways (like eating up memory)
• After each time you add a package or basically anything new is configured, check with htop or top to see how much memory you're using up with the current programs. If you notice that you're using almost all of it, try and clear some out by going through and removing unnecessary programs/processes.
• If there is anything that's being auto-started (besides system processes, of course!) that you don't recognize or want to be auto-started, remove it! But always do your research on what a process is before killing/deleting it, as it could be essential for bootup procedures or system functions, etc.

Solution 2

To get out of this condition without rebooting, you can trigger the OOM killer manually as follows:

echo 1 > /proc/sys/kernel/sysrq
echo f > /proc/sysrq-trigger
echo 0 > /proc/sys/kernel/sysrq

Reference

• How does the OOM killer decide which process to kill first?

Solution 3

In completion to the accepted answer there is one additional thing to consider: Your system may run out of file handles or even socket buffers and still have lots of memory whilst giving the same error. This is especially true if the shared hosting imposes limits of such nature. On OpenVZ systems, watch the contents of

# cat /proc/user_beancounters

This will give you in the rightmost column the first overruns. If this is true, either move to a larger hosting package or hunt down the most likely culprit: the mysql or mariadb database which may, in presence of a defective PHP app, leak file handles to the hundreds per second.

This may also happen if your webserver has ssh open to the internet and accepts username / password logins: even if you have fail2ban running, you may have attracted a distributed dictionary break in attempt which also consumes a lot of resources.

Comments

[Pablo Bianchi]

Do you have any documentation supporting those commands? Why not just sudo sysctl -w vm.oom_kill_allocating_task=1 or permanently on /etc/sysctl.conf.

[Dark]

Doesn't sound like that would make a difference, the system does not hit an actual OOM condition if this happens at rest because no process is trying to allocate memory and no additional processes can be started. And semi-unrelated but you wouldn't be able to use sudo or sysctl once in this state.

[boulder_ruby]

This worked for me. Don't know how, don't care. I wasn't even able to run sudo reboot before running this. Thanks!

[David]

Killing a random process may just kill the whole OS. VERY BAD IDEA.

[M S]

of course not random (i thought that was implied)