"Cannot allocate memory" while no process seems to be using up memory

Question

I am not competent on server issues, any help is much appreciated. When try to start a python/django shell on a linux box, I am getting OSError: [Errno 12] Cannot allocate memory.

free -m seems to confirm I am out of memory:

             total       used       free     shared    buffers     cached
Mem:           590        560         29          0          3         37
-/+ buffers/cache:        518         71
Swap:            0          0          0

But I cannot see what is eating up the memory with top or ps aux:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
    1 root      20   0 24336  908    0 S  0.0  0.2   0:00.68 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      20   0     0    0    0 S  0.0  0.0   0:04.85 ksoftirqd/0        

How can I identify the leak? Thanks.

BTW, I am not sure if it is relevant, but the machine I am talking about is an AWS EC2 instance with Ubuntu 12 running.

Solution 1

Try this:

1. Go into top.
2. Press F (i.e., Shift+F) to specify the sort field.
3. Type N.  This will turn the lowercase n in the list into an uppercase N.
4. Press Enter.  The list should now be sorted by memory usage.

Press ? or h at the main screen for more options.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1010 mysql     20   0  545m  37m 1412 S  0.0  6.3   0:32.97 mysqld
  939 bind      20   0  157m  11m 1896 S  0.0  1.9   0:00.03 named
 3036 ubuntu    20   0 25260 7812 1720 S  0.0  1.3   0:00.42 bash
 1212 root      20   0  192m 7420 2680 S  0.0  1.2   0:03.87 apache2
 1322 www-data  20   0  192m 5880 1088 S  0.0  1.0   0:00.00 apache2
 1323 www-data  20   0  192m 5188  436 S  0.0  0.9   0:00.00 apache2
 1325 www-data  20   0  192m 5188  436 S  0.0  0.9   0:00.00 apache2
 1328 www-data  20   0  192m 5188  436 S  0.0  0.9   0:00.00 apache2
 1329 www-data  20   0  192m 5188  436 S  0.0  0.9   0:00.00 apache2
  968 whoopsie  20   0  197m 4332 2980 S  0.0  0.7   0:00.01 whoopsie
  795 root      20   0  153m 4256 3036 S  0.0  0.7   0:00.06 NetworkManager
 2919 root      20   0 90156 3964 3064 S  0.0  0.7   0:00.02 sshd
  804 root      20   0  101m 3656 2656 S  0.0  0.6   0:00.03 cupsd
 1498 root      20   0  570m 3396 2312 S  0.0  0.6   0:00.03 console-kit-dae
  693 root      20   0 49948 2876 2272 S  0.0  0.5   0:00.01 sshd
  756 root      20   0 79036 2844 2044 S  0.0  0.5   0:00.02 modem-manager
  964 root      20   0  124m 2844 2348 S  0.0  0.5   0:00.02 gdm-binary

Solution 2

ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS

will also sort all processes by memory usage descending.

Comments

[onurmatik]

yes, thank you. it turned out that uwsgi instances got out of control :)

[user1179459]

even for this command i get cannot allocate memory

[Speederer]

@user1179459 Try temporarily adding a swap file, then start debugging what's eating up your memory once you've got a stable system. You can get rid of it once you've fixed the root cause.

[Nakilon]

not enough memory to launch top ..(

[Nakilon]

worked from 10th attempt! so I saw PID even without sorting and then did ssh kill -9