How to set OOM killer adjustments for daemons permanently?

Question

Running some Linux servers with single or just a few vital system service daemons, I would like to adjust the OOM killer for those daemonized processes in case something odd happens. For example, today some Ubuntu server running MySQL got a killed MySQL daemon because tons of apt-checker processes were consuming all memory and the kernel thought it was a good idea to kill MySQL.

I know I can adjust the score using the /proc/$(pidof mysqld)/oom_score_adj file to give the kernel some clue I don't prefer MySQL to be killed, yet that doesn't survive a restart of the service. Should I edit init/upstart scripts from the package to include these adjustments? I don't think that's a very elegant solution as I would make adjustments to files belonging to a package. Would it be possible to hook into upstart/init scripts in general and conditionally adjust it? Or would you suggest running an indefinite script like while true{ adjust_oom(); sleep 60;}?

Solution 1

Several modern dæmon supervision systems have a means for doing this. (Indeed, since there is a chain loading tool for the job, arguably they all have a means for doing this.)

• Upstart: Use oom score in the job file.

  oom score -500

• systemd: Use the OOMScoreAdjust= setting in the service unit. You can use service unit patch files to affect pre-packaged service units.

  [Service]
  OOMScoreAdjust=-500

• daemontools family: Use the oom-kill-protect tool from the nosh toolset in the run program for the service.

  If you are converting a system service unit, the convert-systemd-units tool will in fact convert the OOMScoreAdjust= setting into such an invocation of oom-kill-protect.

  #!/bin/nosh
  …
  oom-kill-protect -- -500
  …
  program arguments

  As a bonus, you can make it parameterizable:

  oom-kill-protect -- fromenv

  and set the value of the parameter in the service's environment (presumed to be read from an envdir associated with the service, here manipulated with the nosh toolset's rcctl shim):

  rcctl set servicename oomprotect -500

Further reading

• Jonathan de Boyne Pollard (2016). oom-kill-protect. nosh toolset. Softwares.
• James Hunt and Clint Byrum (2014). "oom score". Upstart Cookbook.
• Lennart Poettering (2013-10-07). "OOMScoreAdjust". systemd.exec. systemd manual pages. freedesktop.org.
• Jonathan de Boyne Pollard. rcctl. nosh toolset. Softwares.
• https://unix.stackexchange.com/a/409454/5132

Solution 2

This is possible in Ubuntu using Upstart and the oom score configuration option.

oom score

Linux has an "Out of Memory" killer facility. [...]

Normally the OOM killer regards all processes equally, this stanza advises the kernel to treat this job differently.

The "adjustment" value provided to this stanza may be an integer value from -999 (very unlikely to be killed by the OOM killer) up to 1000 (very likely to be killed by the OOM killer). [...]

Example:

# this application is a "resource hog"
oom score 1000

expect daemon
respawn
exec /usr/bin/leaky-app

Solution 3

You could hack it into MySQL itself (e.g. OpenSSH's sshd does this), yet that's a bit too hardcore and very dirty (problems with updates etc.)

You can do this in a wrapper or in the init script - the score should be inherited (and in a wrapper you would probably want to do exec mysqld "$@" anyway).

Use cgroups - it will give you a bit more flexibility and it can be made permanent in the sense, that the appropriate settings can be applied automatically on service restart. See e.g. controlling priority of applications using cgroups for more info. To achieve the automatism you are looking for, you'll probably want to take a look at libcgroup, which contains a daemon that can handle changing cgroups of a running process on the fly according to a set of rules, or just use the cgexec wrapper (from the same package).

Comments

[Nils]

Interesting that there is the possibility to adjust that. I guess that there is nothing better than your infinite loop to accomplish the job. The OOM-killer is buried deep inside the kernel and has a very obscure algorithm.

[gertvdijk]

For readers using Ubuntu 16.04+, this has become obsolete now that Upstart has been replaced by systemd.