Killing a script launched in a Process via os.system()

Question

I have a python script which launches several processes. Each process basically just calls a shell script:

from multiprocessing import Process
import os
import logging

def thread_method(n = 4):
    global logger
    command = "~/Scripts/run.sh " + str(n) + " >> /var/log/mylog.log"
    if (debug): logger.debug(command)
    os.system(command)

I launch several of these threads, which are meant to run in the background. I want to have a timeout on these threads, such that if it exceeds the timeout, they are killed:

t = []
for x in range(10):
    try:
        t.append(Process(target=thread_method, args=(x,) ) )
        t[-1].start()
    except Exception as e:
        logger.error("Error: unable to start thread")
        logger.error("Error message: " + str(e))
logger.info("Waiting up to 60 seconds to allow threads to finish")
t[0].join(60)
for n in range(len(t)):
    if t[n].is_alive():
    logger.info(str(n) + " is still alive after 60 seconds, forcibly terminating")
     t[n].terminate()

The problem is that calling terminate() on the process threads isn't killing the launched run.sh script - it continues running in the background until I either force kill it from the command line, or it finishes internally. Is there a way to have terminate also kill the subshell created by os.system()?

Solution 1

You should use an event to signal the worker to terminate, run the subprocess with subprocess module, then terminate it with Popen.terminate(). Calling Process.terminate() will not allow it worker to clean up. See the documentation for Process.terminate().

Solution 2

In Python 3.3, the subprocess module supports a timeout: http://docs.python.org/dev/library/subprocess.html

For other solutions regarding Python 2.x, please have a look in this thread: Using module 'subprocess' with timeout

Solution 3

Use subprocess instead, whose objects have a terminate() method explicitly for this.

Solution 4

Based on Stop reading process output in Python without hang?:

import os
import time
from subprocess import Popen

def start_process(n, stdout):
    # no need for `global logger` you don't assign to it
    command = [os.path.expanduser("~/Scripts/run.sh"), str(n)]
    logger.debug(command) # no need for if(debug); set logging level instead
    return Popen(command, stdout=stdout) # run directly

# no need to use threads; Popen is asynchronous 
with open('/tmp/scripts_output.txt') as file:
    processes = [start_process(i, file) for i in range(10)]

    # wait at most timeout seconds for the processes to complete
    # you could use p.wait() and signal.alarm or threading.Timer instead
    endtime = time.time() + timeout
    while any(p.poll() is None for p in processes) and time.time() < endtime:
        time.sleep(.04)

    # terminate unfinished processes
    for p in processes:
        if p.poll() is None:
            p.terminate()
            p.wait() # blocks if `kill pid` is ignored

Use p.wait(timeout) if it is available.

View more solutions

Comments

[jfs]

you confuse subprocess.Popen.terminate and multiprocessing.Process.terminate

[quantum]

@J.F.Sebastian But multiprocessing.Process.terminate() doesn't allow the process it controls to clean up. What I mean is to use an event to terminate multiprocessing's worker so it can clean up and call subprocess.Popen.terminate() on the subprocess the worker is controlling.

[jfs]

You're right. I've just ignored that part of the question because the answer doesn't require neither threads (threading) nor processes (multiprocessing).