How to terminate a python subprocess launched with shell=True

Question

I'm launching a subprocess with the following command:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

However, when I try to kill using:

p.terminate()

or

p.kill()

The command keeps running in the background, so I was wondering how can I actually terminate the process.

Note that when I run the command with:

p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)

It does terminate successfully when issuing the p.terminate().

Solution 1

Use a process group so as to enable sending a signal to all the process in the groups. For that, you should attach a session id to the parent process of the spawned/child processes, which is a shell in your case. This will make it the group leader of the processes. So now, when a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.

Here's the code:

import os
import signal
import subprocess

# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before  exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE, 
                       shell=True, preexec_fn=os.setsid) 

os.killpg(os.getpgid(pro.pid), signal.SIGTERM)  # Send the signal to all the process groups

Solution 2

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
p.kill()

p.kill() ends up killing the shell process and cmd is still running.

I found a convenient fix this by:

p = subprocess.Popen("exec " + cmd, stdout=subprocess.PIPE, shell=True)

This will cause cmd to inherit the shell process, instead of having the shell launch a child process, which does not get killed. p.pid will be the id of your cmd process then.

p.kill() should work.

I don't know what effect this will have on your pipe though.

Solution 3

If you can use psutil, then this works perfectly:

import subprocess

import psutil


def kill(proc_pid):
    process = psutil.Process(proc_pid)
    for proc in process.children(recursive=True):
        proc.kill()
    process.kill()


proc = subprocess.Popen(["infinite_app", "param"], shell=True)
try:
    proc.wait(timeout=3)
except subprocess.TimeoutExpired:
    kill(proc.pid)

Solution 4

I could do it using

from subprocess import Popen

process = Popen(command, shell=True)
Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))

it killed the cmd.exe and the program that i gave the command for.

(On Windows)

Solution 5

When shell=True the shell is the child process, and the commands are its children. So any SIGTERM or SIGKILL will kill the shell but not its child processes, and I don't remember a good way to do it. The best way I can think of is to use shell=False, otherwise when you kill the parent shell process, it will leave a defunct shell process.

View more solutions

Comments

[user175259]

In my case it doesn't really help given that cmd is "cd path && zsync etc etc". So that actually makes the command to fail!

[Matt Billenstein]

Use absolute paths instead of changing directories... Optionally os.chdir(...) to that directory...

[mouad]

@PiotrDobrogost: Sadly no, because os.setsid is not available in windows (docs.python.org/library/os.html#os.setsid), i don't know if this can help but you can look here (bugs.python.org/issue5115) for some insight about how to do it.

[Piotr Dobrogost]

How does subprocess.CREATE_NEW_PROCESS_GROUP relate to this?

[mouad]

@PiotrDobrogost: Well found :), apparently if you use the subprocess.CREATE_NEW_PROCESS_GROUP flag you can create a process group in Window check here for how: hg.python.org/cpython/file/321414874b26/Lib/test/… , sadly i don't have a windows machine in my hand so can you please try it and let me know ? :)

[Piotr Dobrogost]

Running python -c "import subprocess; subprocess.Popen(['ping', '-t', 'google.com'], shell=True).terminate()" kills the subprocess. However I think it has nothing to do with subprocess.CREATE_NEW_PROCESS_GROUP as it's not being set anywhere in subprocess.py. Besides according to docs Process groups are used by the GenerateConsoleCtrlEvent function to enable sending a CTRL+BREAK signal to a group of console processes. and Popen.terminate() doesn't send any signal but calls TerminateProcess() Windows API function.

[hwjp]

our testing sugggests that setsid != setpgid, and that os.pgkill only kills subprocesses that still have the same process group id. processes that have changed process group are not killed, even though they may still have the same session id...

[jfs]

@PiotrDobrogost: CREATE_NEW_PROCESS_GROUP can be used to emulate start_new_session=True on Windows

[FDS]

This works even if shell=False. The finding of hwjp still applies.

[parasietje]

I would not recommend doing os.setsid(), since it has other effects as well. Among others, it disconnects the controlling TTY and makes the new process a process group leader. See win.tue.nl/~aeb/linux/lk/lk-10.html

[damjan]

@parasietje: isn't this the whole point of this approach? To create a new process group, in which all processes can be killed with one signal? I had problems with a process that started a new process which I couldn't terminate. This answer solved my problem.

[d33tah]

AttributeError: 'Process' object has no attribute 'get_children for pip install psutil.

[Seng Cheong]

The ability to change the working directory for the child process is built-in. Just pass the cwd argument to Popen.

[MarSoft]

Nice and light solution for *nix, thanks! Works on Linux, should work for Mac as well.

[Godsmith]

I think get_children() should be children(). But it did not work for me on Windows, the process is still there.

[Jovik]

@Godsmith - psutil API has changed and you're right: children() does the same thing as get_children() used to. If it doesn't work on Windows, then you might want to create a bug ticket in GitHub

[peschü]

setsid creates a new session, while setpgrp only creates a new session if there is no session. if you nest several scripts, the outermost should call setsid and all others setpgrp otherwise the inner scripts will again reparent to init and not be killed automatically. en.wikipedia.org/wiki/Process_group#Details

[HelloGoodbye]

How would you do this in Windows? setsid is only available on *nix systems.

[Nicolinux]

Very nice solution. If your cmd happens to be a shell script wrapper for something else, do call the final binary there with exec too in order to have only one subprocess.

[Sergiy Kolodyazhnyy]

This is beautiful. I have been trying to figure out how to spawn and kill a subprocess per workspace on Ubuntu. This answer helped me. Wish i could upvote it more than once

[hungryWolf]

I used shlex, but still the issue persists, kill is not killing the child processes.

[gnr]

this doesn't work if a semi-colon is used in the cmd

[uchuugaka]

Will this fail where os.setsid() will fail with errno.EPERM (errno 1) such as running the script in a GUI terminal session? Or does it work because it happens after fork() but before exec() ?

[alper]

This prints Terminated output, how to prevent it to be printed? @mouad

[user9869932]

This solution doesn't work for me in linux and python 2.7

[epinal]

@xyz It did work for me in Linux and python 3.5. Check the docs for python 2.7

[user9869932]

@espinal, thanks, yes. It's possibly a linux issue. It's Raspbian linux running on a Raspberry 3

[Charlie Parker]

if I don't have shell=True, how much does the answer change?

[DarkLight]

@speedyrazor - Does not work on Windows10. I think os specific answers should be clearly marked as such.

[zvi]

Or use the process name: Popen("TASKKILL /F /IM " + process_name), if you don't have it, you can get it from the command parameter.

[Smak]

this does not work if child X creates child SubX during calling proc.kill() for child A

[jash101]

Can someone tell if this has any bad effects? This does solve the problem.

[Jeff Wright]

Note: this is Wndows-specific. There is no CTRL_C_EVENT defined in Mac or Linux implementations of signal. Some alternative code (which I have not tested) can be found here.

[roshambo]

For some reason the other solutions would not work for me after many attempts, but this one did!!

[rkachach]

couldn't preexec_fn=os.setpgrp be used in this case?

[FriskySaga]

@CharlieParker If shell=False, then I believe you can simply call the terminate() or kill() methods on the Popen object, as the question creator mentioned.

[Jean-François Fabre]

this is a solution that will work on windows too

[Vincent Alex]

For some reason it kills my Putty session when I just type "exec ls" :D