How do you kill all Linux processes that are older than a certain age?

Question

I have a problem with some zombie-like processes on a certain server that need to be killed every now and then. How can I best identify the ones that have run for longer than an hour or so?

Solution 1

If they just need to be killed:

if [[ "$(uname)" = "Linux" ]];then killall --older-than 1h someprocessname;fi

If you want to see what it's matching

if [[ "$(uname)" = "Linux" ]];then killall -i --older-than 1h someprocessname;fi

The -i flag will prompt you with yes/no for each process match.

Solution 2

Found an answer that works for me:

warning: this will find and kill long running processes

ps -eo uid,pid,etime | egrep '^ *user-id' | egrep ' ([0-9]+-)?([0-9]{2}:?){3}' | awk '{print $2}' | xargs -I{} kill {}

(Where user-id is a specific user's ID with long-running processes.)

The second regular expression matches the a time that has an optional days figure, followed by an hour, minute, and second component, and so is at least one hour in length.

Solution 3

For anything older than one day,

ps aux

will give you the answer, but it drops down to day-precision which might not be as useful.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   7200   308 ?        Ss   Jun22   0:02 init [5]
root         2  0.0  0.0      0     0 ?        S    Jun22   0:02 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Jun22   0:18 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Jun22   0:00 [watchdog/0]

If you're on linux or another system with the /proc filesystem, In this example, you can only see that process 1 has been running since June 22, but no indication of the time it was started.

stat /proc/<pid>

will give you a more precise answer. For example, here's an exact timestamp for process 1, which ps shows only as Jun22:

ohm ~$ stat /proc/1
  File: `/proc/1'
  Size: 0               Blocks: 0          IO Block: 4096   directory
Device: 3h/3d   Inode: 65538       Links: 5
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2008-06-22 15:37:44.347627750 -0700
Modify: 2008-06-22 15:37:44.347627750 -0700
Change: 2008-06-22 15:37:44.347627750 -0700

Solution 4

In this way you can obtain the list of the ten oldest processes:

ps -elf | sort -r -k12 | head -n 10

Solution 5

Perl's Proc::ProcessTable will do the trick: http://search.cpan.org/dist/Proc-ProcessTable/

You can install it in debian or ubuntu with sudo apt-get install libproc-processtable-perl

Here is a one-liner:

perl -MProc::ProcessTable -Mstrict -w -e 'my $anHourAgo = time-60*60; my $t = new Proc::ProcessTable;foreach my $p ( @{$t->table} ) { if ($p->start() < $anHourAgo) { print $p->pid, "\n" } }'

Or, more formatted, put this in a file called process.pl:

#!/usr/bin/perl -w
use strict;
use Proc::ProcessTable;
my $anHourAgo = time-60*60;
my $t = new Proc::ProcessTable;
foreach my $p ( @{$t->table} ) {
    if ($p->start() < $anHourAgo) {
        print $p->pid, "\n";
    }
}

then run perl process.pl

This gives you more versatility and 1-second-resolution on start time.

View more solutions

Comments

[yukondude]

Good tip. I stumbled upon the --older-than switch later on, but the -i makes it useful for checking before killing who knows what.

[Rafael S. Calsaverini]

We can join the two commands to get seconds since the process started: "echo stat -t /proc/<pid> | awk '{print $14}' - date +%s | bc"

[pl1nk]

It seems that ps and stat is showing different results for me. ps shows that the process started 1 day ago and stat shows that started today. Why?

[matthias krull]

Note that the TIME column in ps output does not show the actual run time of a process. It shows the accumulated CPU time of the process - the time the CPUs did work with the process.

[Buttle Butkus]

Umm, are you killing the process? I hope people realize this code doesn't just find, but also kills, or they might get upset.

[yukondude]

@ButtleButkus Good point. Yes, the whole reason for the question was to find the old processes and kill them, but it's not explicitly mentioned all that clearly. Note to others: ignore the last little bit of the line unless you enjoy angry calls from users.

[neal aise]

wtf! please change the title of the question. luckily i din't own the process!

[Admin]

This won't always be right -- at least not for Linux 2.6 systems. I have a process that started at 9:49 but stat -t (and stat) show that it started at 13:14.

[baptx]

thanks, stat /proc/<pid> gave me exact result for process lifetime (startup time)

[higuita]

@dpk: sometimes you have a main process and some forks running. The main process should be 9:49, but the child process can have any more recent time. The same applies to the threads of a process.

[Steven Lu]

I'll also note in passing that if the process was started in the previous year, the START column lists just the year. Not very much precision. at all. In addition to this, when I stat'd the proc/id of a long-running tmux session from last year, it reported back a date from this year. Solution I went with: ps -eo pid,etime | grep $PID

[TommyAutoMagically]

What's the point of if [[ "$(uname)" = "Linux" ]];? Isn't the relevant portion just the killall command? (It seems that the surrounding if clause could be removed to make this answer a little more direct)

[Marki555]

You can use option etimes instead of etime to always display the elapsed time in seconds and not days/hours...

[yukondude]

@Marki555 that sounds like just the ticket. Must be in a more recent version of ps that I'm running though, as it's not listed as an option.

[Marki555]

@yukondude you are right. I have just checked it and ps v3.2.8 from debian squeeze does not support the etimes parameter, however v3.3.3 from debian wheezy does.

[Slaven Rezic]

Unfortunately the time output here is difficult to parse. It can be "HH:MM" for short-running processes, or "MonDD" (possibly localized!) or even just the year for very long-running processes.

[ctc]

@ringo Because on some systems (e.g. Solaris), killall is an entirely different command. On Solaris, it terminates all commands.

[Sarel Botha]

Actually that gives you the 10 latest processes because by default it shows STIME which is the time the program started. If it was showing ETIME which is the elapsed time since the program started then this would be correct.

[Jeremy]

Note that using killall's '--regex' option causes '--older-than' to be ignored. Joy!

[user1774301]

Shouldn't $RN be $NR?