Python spawn off a child subprocess, detach, and exit

Question

I'm wondering if this is the correct way to execute a system process and detach from parent, though allowing the parent to exit without creating a zombie and/or killing the child process. I'm currently using the subprocess module and doing this...

os.setsid() 
os.umask(0) 
p = subprocess.Popen(['nc', '-l', '8888'],
                     cwd=self.home,
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT)

os.setsid() changes the process group, which I believe is what lets the process continue running when it's parent exits, as it no longer belongs to the same process group.

Is this correct and also is this a reliable way of performing this?

Basically, I have a remote control utility that communicate through sockets and allows to start processes remotely, but I have to ensure that if the remote control dies, the processes it started continue running unaffected.

I was reading about double-forks and not sure if this is necessary and/or subprocess.POpen close_fds somehow takes care of that and all that's needed is to change the process group?

Thanks.

Ilya

Solution 1

popen on Unix is done using fork. That means you'll be safe with:

1. you run Popen in your parent process
2. immediately exit the parent process

When the parent process exits, the child process is inherited by the init process (launchd on OSX) and will still run in the background.

The first two lines of your python program are not needed, this perfectly works:

import subprocess
p = subprocess.Popen(['nc', '-l', '8888'],
                     cwd="/",
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT)

I was reading about double-forks and not sure if this is necessary

This would be needed if your parent process keeps running and you need to protect your children from dying with the parent. This answer shows how this can be done.

How the double-fork works:

1. create a child via os.fork()
2. in this child call Popen() which launches the long running process
3. exit child: Popen process is inherited by init and runs in the background

Why the parent has to immediately exit? What happens if it doesn't exit immediately?

If you leave the parent running and the user stops the process e.g. via ctrl-C (SIGINT) or ctrl-\ (SIGQUIT) then it would kill both the parent process and the Popen process.

What if it exits one second after forking?

Then, during this 1s period your Popen process is vulnerable to ctrl-c etc. If you need to be 100% sure then use the double forking.

Solution 2

For Python 3.8.x, the process is a bit different. Use the start_new_session parameter available since Python 3.2:

import shlex
import subprocess

cmd = "<full filepath plus arguments of child process>"
cmds = shlex.split(cmd)
p = subprocess.Popen(cmds, start_new_session=True)

This will allow the parent process to exit while the child process continues to run. Not sure about zombies.

The start_new_session parameter is supported on all POSIX systems, i.e. Linux, MacOS, etc.

Tested on Python 3.8.1 on macOS 10.15.5

Comments

[Hubro]

Can you explain why the parent has to immediately exit? What happens if it doesn't exit immediately? Also what constitutes as "immediately"? What if it exits one second after forking?

[hansaplast]

@Hubro: I've added the answers to your question into my answer above and also clarified a few things which were poorly explained

[Dr_Zaszuś]

This process will hang up if the child process puts a large output into pipe. Set stdout=None if you don't want to communicate with the process. For more details see: thraxil.org/users/anders/posts/2008/03/13/…

[Admin]

I can confirm start_new_session works in Python 3.7 and since Python 3.2. In my opinion, this is the cleanest method, at least for me.

[Admin]

I should clarify that it works for me in Debian 10 and Python 3.7.3.

[Saurav Pathak]

start_new_session=True works like charm. If you want to supress output, add these args, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT.

[Vishwanath Heddoori]

What's the cleanest method for python 3.8 onwards?

[Alex Peters]

@VishwanathHeddoori, admittedly a little late in response, but take a look at detach.call.