Prevent Ubuntu from freezing, even if system memory is low

Question

Sometimes I work with huge dumps of data I want to keep in memory for processing. Sometimes I miscalculate the amount of memory my program will produce, or a debugger multiplies the memory usage by a factor that exceeds my available memory.

Whenever I start a memory-hungry process, this is what I'd expect from a sane operating system: try to eat all free memory, then ask some other non-essential processes nicely to give up some memory they don't need, then write to swap.

Here's what Ubuntu does for me: eat all fre memory, then ask the operating system to swap all essential services (gnome session, terminal, keyboard), then freeze and wait for me to pull the power plug.

Two questions:

1. How can an operating system assume, that anything could be so important that it is ok to stop listening to user input?
2. How can I tell Ubuntu to never swap essential services and always react to user input, even if some stupid process tries to eat up more resources than the system provides.

Solution

I still don't have a solution for the problem, but I can offer two workarounds which might be of interest to others:

1) earlyoom

That's a service that watches memory usage and kills the process that consumes the most memory when a certain threshold is reached (see also this and this question regarding the OOM killer in the linux kernel)

I've tested it with a demo process that indefinetly requests memory in small chunks. Here's my first impression: When I start the rogue process, it quickly eats up all of my RAM. Then swapping begins, the system becomes inresponsive. A few seconds later the system is back online. The log of earlyoom shows that it killed the memory eating process after both memory and swap usage reached 90%.

There is still the annoying lag when swapping begins and after the process was killed, some portions of other processes usually remain in swap until they are requested, but it's a start.

2) just disable swap

I know this is a controversial topic, but for the purpose of desktop systems and especially development machines where it can happen from time to time that a process tries to eat up all your memory, it makes sense: Without swap, the OOM killer just works as intended. When you run out of memory, it finds the best process to kill and gets rid of it. No lag, no delay.

You can disable swap for the current session with sudo swapoff -a or make the change permanent.

---

The proper solution for the problem would of course be that the system stays responsive when the main memory is depleted and it starts to swap memory like there's no tomorrow, but that doesn't seem to be happening any time soon.

Comments

[Boris Hamanov]

How much RAM do you have installed? What size is your swap (in terminal, type swapon to find out)? Cheers, Al

[Klamann]

16gb ram and 16gb swap. But that's not the point here, this problem cannot be solved by adding more memory.

[Boris Hamanov]

Try one of two things. 1) change the swappiness setting to 10, ie: vm.swappiness = 10 in /etc/sysctl.conf. Search here for swappiness for more info about it. 2) If swappiness doesn't help... Even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps. Keep me posted. Cheers, Al

[WinEunuuchs2Unix]

@heynnema On my Kernel 4.1.13 version of Ubuntu 14.04 /etc/sysctl.conf doesn't contain the vm.swappiness variable. When booting Kernel 4.7.2 under Ubuntu 16.04 the /etc/sysctl.conf file doesn't even exist and I think I read somewhere that systemd documentation says to use /etc/sysctl.d/ directory's configuration files instead of sysctl.conf. FYI.

[WinEunuuchs2Unix]

@Klamann I agree adding more swap won't fix the problem. Once you have a broken program consuming all RAM+SWAP adding extra SWAP only delays the inevitable.

[Boris Hamanov]

@WinEunuuchs2Unix, my 16.04 system contains /etc/sysctl.conf and /etc/sysctl.d contains a link back to the same file. vm.swappiness needs to be added to sysctl.conf, and rebooted, for it to kick in. In terminal, sysctl vm.swappiness will show that the pre-edit value is probably 70, post-edit should be 10. Since the OP is somewhat reluctant to add physical memory, changing the swap is an easy step to see if he can run without freezing. The general rule of thumb is RAM_SIZE x 1-1.5 for swap... hence I suggested 16G x 1.5. If he has a 2nd HDD on the system, he can even add swap there.

[WinEunuuchs2Unix]

When I type sysctl vm.swappiness the response is 60. But when I cat /etc/sysctl.conf every single line is commented out with # and vm.swappiness parameter doesn't appear there at all. Also under 16.04 but upgraded to Kernel 4.7.2. Later I'll reboot with original kernel 4.4 and update if different.

[Boris Hamanov]

@WinEunuuchs2Unix, as I said, vm.swappiness=10 needs to be ADDED to sysctl.conf. An experienced person could even use the sysctl command on the fly, to set vm.swappiness=10, without editing the sysctl.conf file. Cheers, Al ps: waiting for the OP to respond.

[Boris Hamanov]

Let us continue this discussion in chat.

[谈 超]

did you make any progress on this? I've been also facing this issue for a while and none of the suggestions here help (vm.swappiness=10, vm.min_free_kbytes=12000). I have 16GB of ram and a swap of 4 GB and when they get full, the system freezes. Kernel: 4.10.0-33-generic

[Klamann]

None at all. Adjusting swappiness can slighly reduce the time until the rogue process gets finally killed, but we're still talking about several minutes that the system will be unresponsive, and after you regain control all your applications and your entire OS will be swapped, which will take several more minutes until you can get back to work. So whenever I do memory-intensive work, I'm extremely cautious and if I fuck up, I just reboot the system. Way faster than waiting for a miracle. That's not a solution, but it seems like it's all we've got :/

[George Udosen]

@Klamann Take look at this: oracle.com/technetwork/articles/servers-storage-dev/…

[Klamann]

I don't think the problem can be fixed by adjusting the settings of the OOM killer, because the job of the OOM killer is to destroy processes when nearly all system memory is depleted (including swap). The core issue is that the system becomes unresponsive as soon as swapping begins and after the swap was filled and you're back in control, the system is horribly slow because data is only moved back from swap to main memory on demand. But I finally found a workaround that works for me, after looking into this once more: askubuntu.com/a/960633/572764

[Konrad Gajewski]

2019 and we are still having the same issue. This should not be happening - swap is way slower than ram, but it should not be freezing the machine.

[Klamann]

I've set up a VM to run some tests, because rebooting my OS every few seconds got really annoying. Ubuntu 16.04, 2gb ram, 3gb swap, 20gb disk. Then I ran a script that eats up a lot of memory: With default swappiness (60), the system freezes and after a few minutes, I shut it down because the time to recovery was inacceptible. With swappiness 10, the system freezes for a few seconds, then it accepts input but you can't start any processes (e.g. top to kill the memory hog). After a minute or so, the process gets killed. Not perfect, but we're getting closer.

[Boris Hamanov]

Keep us posted. The VM won't truly emulate your real life working OS, but it'll let you play with the settings. I'll be curious if swappiness helps with your problem. Cheers, Al

[Boris Hamanov]

Ah, good. Progress! Read up a little on swappiness. You can play with the value a little bit. Cheers, Al

[Boris Hamanov]

How much swap was being used when the system froze? Cheers, Al

[Klamann]

none at all, when my script eats up about 1.75gb of my 2gb memory (I guess this is the point when swapping begins), it freezes instantly and depending on the swappines value, it may react to me hitting ctrl+c after a while or not.

[Boris Hamanov]

Interesting. If you're not already doing it, run top in terminal while you're testing, and watch the mem and swap values. Try and play with swappiness. Try values of 0 or 1 or 80. Keep us posted. Cheers, Al

[Klamann]

So here's my thoughts on swappiness: The default value (60) sucks. 10 was ok. 20 was pretty much the same as 10. 0 was slower than 10. But even with a swappiness of 10, I have a long period of time where my system doesn't respond at all and I have to be careful to be able to kill my process with a single keystroke (the system hardly reacts to buttons) or I may have to wait until the OOM killer wakes up, which can take several minutes. All in all, this still blocks the system for too long and the reaction to user input is way too slow.

[Boris Hamanov]

Change the VM RAM to 4G and see what happens. Cheers, Al

[Klamann]

why, what's the point in that? more memory wont prevent the system from getting stuck if I suck up that memory too.

[Boris Hamanov]

Changing the swappiness value changes the dependance on RAM vs swap. With a value of 10, and more RAM, the system will either: 1) handle the low RAM situation better, or 2) at least allow your test script to run longer (and maybe allow your real life app to be able to complete before freezing the system). We may need to take this conversation to chat. Please vote/accept my answer if you think that it helped. Cheers, Al

[Michael]

I have disabled swap and my system goes straight from low memory (<100MB) to freezing. How can I tell if the OOM killer is actuall enabled?

[Konrad Gajewski]

Please expand the links into answers.