Killing a subprocess including its children from python

Question

I'm using the subprocess module on python 2.5 to spawn a java program (the selenium server, to be precise) as follows:

import os
import subprocess

display = 0
log_file_path = "/tmp/selenium_log.txt"
selenium_port = 4455
selenium_folder_path = "/wherever/selenium/lies"

env = os.environ
env["DISPLAY"] = ":%d.0" % display
command = ["java", 
           "-server",
           "-jar", 
           'selenium-server.jar',
           "-port %d" % selenium_port]
log = open(log_file_path, 'a')
comm = ' '.join(command)
selenium_server_process = subprocess.Popen(comm,
                                           cwd=selenium_folder_path,
                                           stdout=log,
                                           stderr=log,
                                           env=env,
                                           shell=True)

This process is supposed to get killed once the automated tests are finished. I'm using os.kill to do this:

os.killpg(selenium_server_process.pid, signal.SIGTERM)
selenium_server_process.wait()

This does not work. The reason is that the shell subprocess spawns another process for java, and the pid of that process is unknown to my python code. I've tried killing the process group with os.killpg, but that kills also the python process which runs this code in the first place. Setting shell to false, thus avoiding java to run inside a shell environment, is also out of the question, due to other reasons.

How can I kill the shell and any other processes generated by it?

Solution 1

To handle the general problem:

p=subprocess.Popen(your_command, preexec_fn=os.setsid)
os.killpg(os.getpgid(p.pid), signal.SIGTERM)

setsid will run the program in a new session, thus assigning a new process group to it and its children. calling os.killpg on it thus won't bring down your own python process also.

Solution 2

The obvious solution in this case is to not involve the shell:

import os
import subprocess

display = 0
log_file_path = "/tmp/selenium_log.txt"
selenium_port = 4455
selenium_folder_path = "/wherever/selenium/lies"

env = os.environ
env["DISPLAY"] = ":%d.0" % display
command = ["java", 
           "-server",
           "-jar", 
           'selenium-server.jar',
           "-port",
           str(selenium_port)]
log = open(log_file_path, 'a')
selenium_server_process = subprocess.Popen(command,
                                           cwd=selenium_folder_path,
                                           stdout=log,
                                           stderr=subprocess.STDOUT,
                                           env=env)

This will make the process be the Java process directly. Keep in mind that it may still spawn processes that are not part of the process group, so os.killpg may still not know about killing them.

If you have a reason to invoke the shell (the above code does not, and there are few things you cannot do without the shell, but suppose you do), you would have to make the shell pass you the pid of the process it started somehow. Doing this is not straightforward, and rather situational.

Comments

[moshez]

Or, if you do involve the shell for any reason (let's say you want expansion/substitution/whatever), use "exec" at the beginning to avoid the shell forking. (Specifically, just adding "exec" at the beginning of "command" definition would have fixed your problem.)

[Admin]

@moshez, awesome! that did it. thanks a lot.

[Thomas Wouters]

Do you actually have a reason for involving the shell, though? There very rarely is.

[jfs]

No need to use setsid; you can call os.setsid in Python

[berdario]

No, you cannot... that will change the session of the process itself, if what you're after is killing only the children it isn't what you want

[jfs]

reread the title of the question: "Killing a subprocess including its children from python"

[jfs]

fyi, preexec_fn is executed after fork (in the child process)

[berdario]

I've read the title, have you? Thomas' answer didn't deal with killing the children of the subprocess (btw yes, prexec_fn is interesting, thanks)

[jfs]

the title says: process + children. You say: "only children".

[berdario]

the title says "subprocess" + children, I say "children" (meaning, subprocess+children), you say "the calling process itself + children (subprocess and its children)"

[TangHongWan]

Just tried, seems no need preexec_fn=os.setsid. My python is 3.6.8

[berdario]

If you don't preexec_fn=os.setsid , your Python script is going to also kill the program that started the program that started the children. (i.e. the Python script itself in my short snippet)