Run a process and kill it if it doesn't end within one hour

Question

I need to do the following in Python. I want to spawn a process (subprocess module?), and:

• if the process ends normally, to continue exactly from the moment it terminates;
• if, otherwise, the process "gets stuck" and doesn't terminate within (say) one hour, to kill it and continue (possibly giving it another try, in a loop).

What is the most elegant way to accomplish this?

Solution 1

The subprocess module will be your friend. Start the process to get a Popen object, then pass it to a function like this. Note that this only raises exception on timeout. If desired you can catch the exception and call the kill() method on the Popen process. (kill is new in Python 2.6, btw)

import time

def wait_timeout(proc, seconds):
    """Wait for a process to finish, or raise exception after timeout"""
    start = time.time()
    end = start + seconds
    interval = min(seconds / 1000.0, .25)

    while True:
        result = proc.poll()
        if result is not None:
            return result
        if time.time() >= end:
            raise RuntimeError("Process timed out")
        time.sleep(interval)

Solution 2

There are at least 2 ways to do this by using psutil as long as you know the process PID. Assuming the process is created as such:

import subprocess
subp = subprocess.Popen(['progname'])

...you can get its creation time in a busy loop like this:

import psutil, time

TIMEOUT = 60 * 60  # 1 hour

p = psutil.Process(subp.pid)
while 1:
    if (time.time() - p.create_time()) > TIMEOUT:
        p.kill()
        raise RuntimeError('timeout')
    time.sleep(5)

...or simply, you can do this:

import psutil

p = psutil.Process(subp.pid)
try:
    p.wait(timeout=60*60)
except psutil.TimeoutExpired:
    p.kill()
    raise

Also, while you're at it, you might be interested in the following extra APIs:

>>> p.status()
'running'
>>> p.is_running()
True
>>>

Solution 3

I had a similar question and found this answer. Just for completeness, I want to add one more way how to terminate a hanging process after a given amount of time: The python signal library https://docs.python.org/2/library/signal.html

From the documentation:

import signal, os

def handler(signum, frame):
    print 'Signal handler called with signal', signum
    raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0)          # Disable the alarm

Since you wanted to spawn a new process anyways, this might not be the best soloution for your problem, though.

Solution 4

A nice, passive, way is also by using a threading.Timer and setting up callback function.

from threading import Timer

# execute the command
p = subprocess.Popen(command)

# save the proc object - either if you make this onto class (like the example), or 'p' can be global
self.p == p

# config and init timer
# kill_proc is a callback function which can also be added onto class or simply a global
t = Timer(seconds, self.kill_proc)

# start timer
t.start()

# wait for the test process to return
rcode = p.wait()

t.cancel()

If the process finishes in time, wait() ends and code continues here, cancel() stops the timer. If meanwhile the timer runs out and executes kill_proc in a separate thread, wait() will also continue here and cancel() will do nothing. By the value of rcode you will know if we've timeouted or not. Simplest kill_proc: (you can of course do anything extra there)

def kill_proc(self):
    os.kill(self.p, signal.SIGTERM)

View more solutions