nice, ionice are not enough

Question

I have a script that starts big, CPU and memory consuming tree of processes. There is Python and executables down there, but everything starts with single bash script and python subprocesses.

During the execution, rest of system is completely chocked down. I attempted to do mitigate by $ nice -n10 ionice -c2 ./Script.sh, however this is not sufficient - using computer is very laggy (acutally this is development desktop, but the problem on designated server will be the same analogous).

I suspect, that the problem is with processes using to much memory - everything ends up swapped out and becomes sluggish.

Is there a way to lower priority of a process (and its recursive children) in access to physial memory? I prefer it to be done slower in background with limited influence on other tasks.

Solution 1

You cannot limit "pace" of consuming memory, but you can limit it's total memory usage via various different mechanisms.

1) security limits Limit memory usage for user running the process via /etc/security/limits.conf. This may not work in your case if you are running this process as the same user working on different stuff.

Example:

username hard as 1000000

2) Control Groups You can - with cgroups, create a group and also limit the memory usage. Just create cgroup, like this:

# cat >> /etc/cgconfig.conf << EOF
group memlimit {
    memory {
        memory.limit_in_bytes = 1073741824;
    }
}
EOF

# cat >> /etc/cgrules.conf <<EOF
username memory   memlimit/
EOF

Offcourse - in both cases, you have to develop your program so that it can recover from failing to allocate more memory.

In case it can't, you just need to add more memory to your system, so that you can avoid swapping. Once the swapping starts, its in the hands of kernel, and you can't - for example - lower the priority of kswapd, and even if you could - that doesn't guarantee some of the programs you use wouldn't still get swapped out, thus causing even more slower system response. Just don't go there.

Solution 2

While the next will not help you on memory swapping, it should help you on IO impact of your process.

It seems that you should explicit set the level as well.

ionice -c2 -n5 ./slowscript.sh

C2 alone might not be enough, depending on your kernel.

Qoute from the manpage (man ionice)

          Note that before kernel 2.6.26 a process that has not asked for an I/O priority formally uses "none" as scheduling class, but the I/O scheduler will  treat  such
          processes  as if it were in the best-effort class.  The priority within the best-effort class will be dynamically derived from the CPU nice level of the process:
          io_priority = (cpu_nice + 20) / 5.

          For kernels after 2.6.26 with the CFQ I/O scheduler, a process that has not asked for an I/O priority inherits its CPU scheduling class.   The  I/O  priority  is
          derived from the CPU nice level of the process (same as before kernel 2.6.26).

Basically: every newly started process will get C2 N4 , so when you want to have the IO reduced to as low as possible, either go on idle only (C3) or C2 N7.

Comments

[Dan]

Once you're out of memory and start swapping, it's game over. Optimize your program or buy enough memory.

[MateuszL]

Yes, that's true. But I wish it could be game over for one process (process-tree), not for the rest of the system.

[Dan]

Memory is cheap these days

[MateuszL]

Thanks. I searched more on cgroup rules and ended up with: nice -n 10 ionice -c2 -n7 cgexec -g memory:memlimit ./Script.sh

[MateuszL]

I have recent kernel and usually combined ionice with nice -n10 or -n15, so io_priority by default was 6 or 7. Thank You none the less.

[mtraceur]

@MateuszL If your system is memory constrained and bottle-necking on swap though, I have found that ionice -c 3 is a lifesaver. Yeah, the process will just wait for everything else to finish, but assuming your swap partition and the used filesystem partition(s) are on the same physical device or just same bus to the CPU/RAM, you can really make a difference even in situations where swapping is the bottleneck.

[mtraceur]

@MateuszL In particular, I'm talking running full from-source compiles of large projects like GCC and node.js, on a single-core 600MHz ARMv7l chip with 256 MiB RAM. Swapping + IO happens over the same slow bus to the same eMMC chip, and it's the bottleneck. Running builds with ionice -c 3 (even before throwing in a nice) turned the builds from grinding the device into a basically unusable state (up to a worst-case of automatic reboot from watchdog not being pinged often enough), into the device basically remaining casually usable through-out the build.