Limit total memory usage for multiple instances of systemd service

Question

I'm running some .NET Core processes as systemd services under Ubuntu 16.04 (soon to be 18.04). I have a systemd configuration file (/lib/systemd/system/[email protected]) that looks like this:

[Unit]
Description=My Service %i

[Service]
Type=simple
User=myservice
EnvironmentFile=/etc/environment
WorkingDirectory=/home/myservice/instances/%i
ExecStart=/opt/dotnet/dotnet My.Service.dll

[Install]
WantedBy=multi-user.target

I want to limit the total amount of RAM that all instances of this service can use. I don't want to limit the RAM for any individual instance to less than that, so settings like MemoryHigh and MemoryMax aren't helpful.

I know that systemd creates a cgroup for the service template, so I want to change the memory limit for that cgroup somehow.

On Ubuntu 18.04 I can manually edit /sys/fs/cgroup/memory/system.slice/system-myservice.slice/memory.limit_in_bytes and this basically does what I want (processes get killed when the total memory usage exceeds the limit), but there are some issues with this approach:

• This file doesn't always exist on boot until the service is started.
• On Ubuntu 18.04 this file gets overwritten whenever systemctl daemon-reload is called.
• Trying to write to the file sometimes returns write error: Device or resource busy

(Under Ubuntu 16.04 the limit seems to be reset whenever my service starts, so it has no effect.)

Is there some way to get systemd itself to set this value, so I don't have to fight against it? Or some other way to limit the total memory use for a group of processes? They all run as the same user, so it would be OK to limit the RAM used by that user, for example.

I even tried manually creating a cgroup (cgcreate -t myservice:myservice -g memory:mycgroup) and then changing ExecStart in the service configuration to /usr/bin/cgexec -g memory:mycgroup /opt/dotnet/dotnet My.Service.dll and this again sort of works, but not reliably: the memory limit I wrote to memory.limit_in_bytes got reset at some point and I don't know when or why.

Solution 1

I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
MemoryHigh=500M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/[email protected]

It was not necessary to set systemd.unified_cgroup_hierarchy=1 (as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.

Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)

Solution 2

We maybe need to use "unified cgroup hierarchy" as explained at Unified and Legacy Control Group Hierarchies .

To enable this feature, add systemd.unified_cgroup_hierarchy=1 to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, run update-grub, and reboot Linux.

systemd.unified_cgroup_hierarchy is explained insystemd unified cgroup hierarchy.

Then added the following lines at [Service] section of your systemd unit file, and run systemctl daemon-reload:

Delegate=memory
MemoryHigh=8G (if you choose 8 gigabytes as the limit)

Explanation of "MemoryHigh" is given at systemd.resource-control.

Comments

[EM0]

Thanks a lot for your help! Even though it was not necessary (in my case) to enable the unified cgroup hierarchy your answer pointed me in the right direction (see my own answer).

[EM0]

Also, a note for anyone trying to enable unified cgroup hierarchy on Ubuntu 18.04: the file to edit may be /etc/default/grub.d/50-cloudimg-settings.cfg or some other file depending on your installation - these files override /etc/default/grub - see bugs.launchpad.net/ubuntu/+source/grub2/+bug/1569567

[user2948306]

Great :-). Template service units should get put in a slice by default already. E.g. [email protected] will get put in system-getty.slice (under system.slice). Might be a useful convention to follow.

[EM0]

They do - but if I set MemoryMax in systemd for the template service unit that sets a per-instance limit, not the total limit for all instances. If I try to set the cgroup limits manually systemd overwrites them, as my question says. A custom slice allows me to set the total limits for the group of processes using systemd.

[EM0]

Oh, I see - in system-myservice.slice, not [email protected] ? That may work, I didn't think to try it.