Python programs suddenly get killed

Question

I'm running some python programs that are quite heavy. I've been running this script for several weeks now, but in the past couple of days, the program gets killed with the message:

Killed

I tried creating a new swap file with 8 GB, but it kept happening.

I also tried using:

dmesg -T| grep -E -i -B100 'killed process'

which listed out the error:

[Sat Oct 17 02:08:41 2020] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected],task=python,pid=56849,uid=1000
[Sat Oct 17 02:08:41 2020] Out of memory: Killed process 56849 (python) total-vm:21719376kB, anon-rss:14311012kB, file-rss:0kB, shmem-rss:4kB, UID:1000 pgtables:40572kB oom_score_adj:0

I have a strong machine and I tried also not running anything else when running ( Pycharm or terminal) but it keeps happening.

specs:

• Ubuntu 20.04 LTS (64bit)
• 15.4 GiB RAM
• Intel Core i7-105100 CPU @ 1.80 GHz x 8

when running free -h t

             total        used        free      shared  buff/cache   available
Mem:           15Gi       2.4Gi        10Gi       313Mi       2.0Gi        12Gi
Swap:         8.0Gi       1.0Gi       7.0Gi

Solution 1

There's nothing to be done here, I'm afraid. The process is being killed by the OOM killer (Out Of Memory Killer), which is a process of the operating system whose job it is to kill jobs that are taking up too much memory before they crash your machine. This is a good thing. Without it, your machine would simply become unresponsive.

So, you need to figure out why your python script is taking up so much memory, and try to make it so that it uses less.

The only other alternative is to try and get more swap, or more RAM of course, but that feels like a bandaid. If this is your python script, you should focus on making it less memory hungry if at all possible.

Solution 2

May be buffer memory got increased since you are running the program for more days. You can clean the buffer memory using the below garbage collection function. Also, you can add gc.collect() wherever required.

 import gc

 gc.collect()

Comments

[terdon]

How exactly did you create the swap file? Did you also activate it? Please edit your question and add the exact commands you ran. Also show us the output of free -h that indicates the swapfile is in use.

[terdon]

You need to check the output of free while your python script is running. You will see that it is using all of the available RAM and also the swap. If so, all you can do is fic the script or give it more ram/swap.