How to limit a systemd service to "play nice" with the CPU?

Question

I have a systemd service (a CI runner) that tends to bog the system down with very CPU intensive jobs. I caught the load average flying over 100 just now and want to put a stop to that nonsense.

Nothing else on the system is limited in any way, so the deal is that I want everything else to continue as it would run now, but either:

• (a) every process running as the unique user the CI jobs run as or
• (b) any child processes instantiated by the systemd service daemon

... to play second fiddle to everything else on the system. It fact I'd like them to have something like a 90% absolute cap even when nothing else on the system needs the remaining 10% of CPU cycles, but if anything else at all requests CPU time I would like them to get as much as they want first.

What is the best way to configure this? I'm running Arch Linux on EC2 and have cgroups available (including cgmanager) but have never used them.

Solution

First of all, most of what comes up in search on the web has been deprecated. For example cgmanager is no longer supported on new systemd versions. Don't follow 99% of what comes up in web serches as far as using cuplimit, nice, cgset or other tools for this job. They either won't work at all as advertised (as in the case of cgroup management tools that expect you to create your own hierarchy) or won't get the job done without resorting to lots of hacks (as in the case of using 'nice' levels to manage whole groups of processes).

The good news is that along with those deprecations (and pursuing the traditional all-devouring octopus monster modus operandi of systemd) a default configuration is in place for everything on the system, and tweaking it for systemd services is trivial. Just add an overlay configuration to the service you want to limit:

$ sudo systemctl edit <servicename>

Add a section with whatever resource control values you want to override. In my case I came up with this:

[Service]
CPUWeight=20
CPUQuota=85%
IOWeight=20
MemorySwapMax=0

These values are not all necessary, but the first two answer the question as asked:

• CPUWeight defaults to 100 for all processes on the system. Setting a low value still lets process use the CPU if nothing else is effectively keeping the system responsive for other tasks while not slowing down the results much. This is a arbitrary weight integer.
• CPUQuota is an absolute limit on how much CPU time is granted even if nothing else is going on. This is a percent value. In my case it wasn't really necessary to set this to fix the resource hogging issue. I ended up setting it anyway to keep the CPU temperature down when lots of CI jobs pile up.
• IOWeight is much the same as CPUWeight, in this case used to keep disks free for system tasks and only keep them busy with CI jobs when nothing else is going on.
• MemorySwapMax also isn't in scope for the question, in my case I ended up adding it because the ray traver (povray) running in some of the CI jobs seems to think using 30+ Gigs of swap in addition to the 30+ Gigs of RAM in this system is good idea just because it is there. It runs faster if you don't let it use it at all. This is probably something better configured in povray, but this way I don't have to police what happens inside of CI jobs and don't have to disable the system swap.

Lastly these values can be changed on the fly without restarting services by running systemctl daemon-reload. This is quite handy to watch the effect of the changes right away.

Comments

[Gen.Stack]

Excellent. Thank you. Years ago I struggled to restrain clamav from hogging CPU and RAM while using the tools available at that time. Namely cgroups and consorts. That was before the rise of systemd, for which I am grateful. Since systemd so many things are configurable consistently and reliably. At least from my perspective.I just works (better and better).

[Admin]

On my system (running systemd v250) that systemctl daemon-reload step was not needed but was done automatically, although people report that it's still needed. Anyway, thanks for that answer!