Automatically kill processes that over time uses 95%+ of resources? Ubuntu

Question

I don't know about your computer but when mine is working properly no process is sucking 95%+ over time. I would like to have some failsafe that kills any processes behaving like that. This comes to mind because when I woke up this morning my laptop had been crunching all night long on a stray chromium child process.

This can probably be done as a cron job, but before I make it a full time job creating something like this I'd thought I should check here. :) I hate reinventing the wheel.

Solution 1

Others have encountered this problem, and while there doesn't seem to be any prevention mechanisms builtin to Ubuntu, there are some ideas about how to improve this.

There's a similar Serverfault question that mentions Monit might be able to help.

Solution 2

I have created a script, kill-process, that kills some processes listed in an array, if CPU usage is greater than XX% for YY seconds or kill processes that's running more than ZZ seconds.

• You can set XX, YY, ZZ in the top of the file.
• You can use a ps or top for check processes.
• There's a dry run mode too, to check but not kill.
• In the end, the script sends an email if some processes were killed.

NOTE: Here is my repo on Github: https://github.com/padosoft/kill-process

Essential part of script (a code abstract for top command):

#!/usr/bin/env bash

#max cpu % load
MAX_CPU=90
#max execution time for CPU percentage > MAX_CPU (in seconds 7200s=2h)
MAX_SEC=1800
#sort by cpu
SORTBY=9

#define a processes command name to check
declare -a KILLLIST
KILLLIST=("/usr/sbin/apache2" "/usr/bin/php5-cgi")

#iterate for each process to check in list
for PROCESS_TOCHECK in ${KILLLIST[*]}
do

    #retrive pid with top command order by SORTBY
    PID=$(top -bcSH -n 1 | grep $PROCESS_TOCHECK | sort -k $SORTBY -r | head -n 1 | awk '{print $1}')

    CPU=$(top -p $PID -bcSH -n 1 | grep $PROCESS_TOCHECK | sort -k $SORTBY -r | head -n 1 | awk '{print $9}')
    TIME_STR=$(top -p $PID -bcSH -n 1 | grep $PROCESS_TOCHECK | sort -k $SORTBY -r | head -n 1 | awk '{print $11}')

    # Decode the top CPU time format [dd-]hh:mm.ss.
    TIME_SEC=0
    IFS="-:" read c1 c2 c3 c4 <<< "$TIME_STR"

    #with top command time format is hh:mm.ss, so truncare seconds in c2
    c2=${c2%%.*}

    if [ -n "$c4" ]
    then
      TIME_SEC=$((10#$c4+60*(10#$c3+60*(10#$c2+24*10#$c1))))
    elif [ -n "$c3" ]
    then
      if [ "$CMD" = "ps" ]; then
        TIME_SEC=$((10#$c3+60*(10#$c2+60*10#$c1)))
      else
        TIME_SEC=$(((10#$c3*24)*60*60)+60*(10#$c2+60*10#$c1))             
      fi   
    else
      if [ "$CMD" = "ps" ]; then
        TIME_SEC=$((10#0+(10#$c2+60*10#$c1)))
      else
        TIME_SEC=$((10#0+60*(10#$c2+60*10#$c1)))
      fi
    fi

    #check if need to kill process
    if [ $CPU -gt $MAX_CPU ] && [ $TIME_SEC -gt $MAX_SEC ]; then
        kill -15 $PID
    fi

done

bash killprocess.sh [dry|kill|--help] [top|ps] [cpu|time]

Comments

[Jjames]

This is for sure possible (using ps and similar tools). But the real question is, do you really want that? Check-Programs of all kind can grind pretty high cpu usage for quiet some time, also image-processing can take that, too. You should ask yourself a question: Is it worth the aftermath if it kills the wrong process?

[Dennis Williamson]

You almost always should fix the underlying problem. It's not really a good idea to blindly kill processes based on their resource usage. That's what the kernel's OOM-killer is for (at least for memory).

[hotei]

I'd also have to argue against doing it with any kind of auto-kill script/job. Maybe auto-notify is the right way and you can decide to kill or not. Fix underlying problem once it's identified. If your computer can send email to your phone then you can do this "real-time" or close enough.

[Lorenzo Padovani]

@DavidPostill I add an abstract of script code. Thank's.

[DavidPostill]

Much better ... :)