MongoDB getting OOM killed

Question

we are running a mongodb replicaset on three machines. All three machines have around 16GB but only 255MB Swap. Swappiness is left on it's default value 60. The machines are running CentOS 6.4. The databases are much larger than the 16GB, but that's ok for us. The really working set is much smaller.

The problem we are facing is that the primary consumes eats up all available memory and than getting OOM-Killed. I know that this is the way how mongodb manages memory.

After the server is getting OOM killed someone has to manually restart it.

Is there any way to prevent mongodb from getting OOM killed? Adjust the swappiness? Increase swap space? I think that those settings will only increase the grace period before mongod gets killed.

Solution

OOM killer is not a way anyone manages memory; it is Linux kernels way to handle fatal failure in last hope to avoid system lockup!

What you should do is:

• make sure you have enough swap. If you are sure, still add more.

• implement resource limits! At LEAST for applications you expect that will use memory (and even more so if you don't expect them to - those ones usually end up being problematic). See ulimit -v (or limit addressspace) commands in your shell and put it before application startup in its init script. You should also limit other stuff (like number of processes -u, etc)... That way, application will get ENOMEM error when there is not enough memory, instead of kernel giving them non-existent memory and afterwards going berserk killing everything around!

• tell the kernel not to overcommit memory. You could do:

  echo "0" > /proc/sys/vm/overcommit_memory

  or even better (depending on your amount of swap space)

  echo "2" > /proc/sys/vm/overcommit_memory; echo "80" > /proc/sys/vm/overcommit_ratio

  See Turning off overcommit for more info on that.

  That would instruct kernel to be more carefull when giving applications memory it doesn't really have (similarity to worlds global economic crisis is striking)

• as a last dich resort, if everything on your system except MangoDB is expendable (but please fix two points above first!) you can make lower the chances of it being killed (or even making sure it won't be killed - even if alternative is hangup machine with nothing working) by tuning /proc/$pid/oom_score_adj and/or /proc/$pid/oom_score.

  echo "-1000" > /proc/`pidof mangod`/oom_score_adj

  See Taming the OOM killer for more info on that subject.

Comments

[Chris Winslett]

Are you running 2.4.0 - 2.4.2?

[bjoernhaeuser]

Sorry, I dont want to limit the memory usage, I just want to prevent OOM :)

[bjoernhaeuser]

Thanks for this thorough explanation of the topic. I can just take one recommendation out of it: increae the swap space. Correct? Your explanation covers this topic on a very wide approach, but I think my problem is very mongodb specific.

[Matija Nalis]

No, not really. You see, OOM is not triggered when application asks for memory and there is none left (in such case, something would get swapped-out or application would get -ENOMEM error), but when kernel needs memory (for example, for allocating memory for TCP/IP packet) and there isn't any available. And while you can often avoid that by increasing amount of swap space and vm.swappiness, it is better to make sure kernel doesn't overcommit so it will have enough memory. And limiting mongodb (and other processes) will make sure they get ENOMEM (and handle it cleanly) instead of kernel OOMing.

[bjoernhaeuser]

MongoDB does not manages memory, it just mmap files. Overcommit is already deactivated.

[Matija Nalis]

I know that MongoDB does mmap(2) instead of brk(2) for databases, but mmap also needs memory to page-in the data. If free memory is not available when new region of mmaped data is accessed, kernel will need to page-out (or swap-out, but not in MongoD dbs) some data. Note that this will not case OOM by itself (unless swap-out is needed and all swap is used up). OOM will be caused if all memory is used, and kernel needs and can't get some without paging/swapping out to disk. I assume that you're not running mongod as your init process, so your other processes might be the triggers: hence advices

[bjoernhaeuser]

A Puppet run triggers OOM