How to get PID via subprocess.Popen with custom environment variable?

Question

Using Python, how can I run a subprocess with a modified environment variable and get its PID? I assume subprocess.Popen() is along the right track...

In shell (bash), I would do this:

MY_ENV_VAR=value ./program_name arg1 arg2 etc &

This runs program_name in the background, passing in "arg1" and "arg2" and "etc", with a modified environment variable, "MY_ENV_VAR" with a value of "value". The program program_name requires the environment variable MY_ENV_VAR to be set to the proper value.

How can do the equivalent thing in Python? I absolutely need the PID of the process. (My intent is to keep the python script running and performing checks on some of the things program_name is doing in the meantime, and I need the process ID to make sure it's still running.)

I've tried:

proc = subprocess.Popen(['MY_ENV_VAR=value', './program_name', 'arg1', 'arg2', 'etc'])

But of course, it expects the first item to be the program, not an environment variable.

Also tried:

environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc', env=environ])

Close, I suppose, but no cigar. Similarly, this:

environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ)

This echoes "$MY_ENV_VAR" literally, I suppose because there's no shell to interpret it. Okay, so I try the above but with this line instead:

proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ, shell=True)

And that's fine and dandy, except that the value that's echoed is blank (doesn't apparently exist). And even if it did work, I'd get the PID of the shell, not the actual process I'm trying to launch.

I need to launch a process with a custom environment variable and get its PID (not the PID of the shell). Ideas?

Solution 1

Your last version is very close, but not quite there.

You don't want $MY_ENV_VAR to be an argument to echo. The echo program will have MY_ENV_VAR in its environment, but echo doesn't do any env variable expansion. You need it to be expanded by the shell, before it even gets to echo.

This may actually have nothing to do with your real-life test case. You already are getting the environment variable to the child process in all of your tests, it's just that echo doesn't do anything with that environment variable. If your real program just needs the environment variable to be set, you're done:

proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc'], env=environ)

But if your program needs it to be substituted, like echo, then you have to substitute it into the arguments before they get passed to your program.

The easiest way to do that is to just give the shell a command line instead of a list of arguments:

proc = subprocess.Popen('echo "$MY_ENV_VAR"', env=environ, shell=True)

People will tell you that you should never use a command string in subprocess—but the reason for that is that you always want to prevent the shell from expanding variables, etc., in a way that could be insecure/etc. On the rare occasions when you want the shell to do its shelly things, you want a command string.

Of course if you use a shell, on most platforms, you're going to end up getting the PID of the shell rather than the PID of the actual program. Short of doing some platform-specific digging to enumerate the shell's children (or wrapping the whole thing in some simple sh code that gives you the child's PID indirectly), there's no way around that. The shell is what you're running.

Another alternative is to expand the variables in Python instead of making the shell do it. Then you don't even need a shell:

proc = subprocess.Popen(['echo', os.path.expandvars('$MY_ENV_VAR')])

… or, even more simply:

proc = subprocess.Popen(['echo', os.environ['MY_ENV_VAR']])

Solution 2

here's a program that spits out the current environment.

#!/usr/bin/env python
##program_name
import os
for k,v in os.environ.iteritems():
    print k, '=', v

Here's a program that calls the other program, but first changes the environment

#!/usr/bin/env python
import subprocess, os
newenv = os.environ.copy()
newenv['MY_ENV_VAR'] = 'value'
args = ['./program_name', 'arg1', 'arg2', 'etc']
proc = subprocess.Popen(args, env=newenv)
pid = proc.pid
proc.wait()
print 'PID =', pid

Comments

[Charles Duffy]

'echo "$MY_ENV_VAR"', please -- otherwise, the variable's contents are string-split and glob-expanded.

[Matt]

Cool, that (first one) works, and I now understand why -- but I get the PID of the shell (sh) instead of the echo process. As for expanding the variable with Python, that's a great idea, but I probably wasn't clear: the program that's being called expects that environment variable to be set, I can't just pass it in as an argument.

[abarnert]

@Matt: You may be reading an older version of the answer. To get the environment variable set, rather than substituted, just use the env parameter, and don't use the shell at all. The only reason that didn't work in your echo tests is that echo needs the variable substituted, not set.

[Matt]

@abarnert You're right. I got your updated answer and Charles' comment at the same time. This appears to do the trick. Thanks! I didn't realize that the environment was passed straight to the program instead of a shell (duh, on my part).

[abarnert]

Why would you do {k:v for k,v in anything.iteritems()} instead of just dict(anything)?

[Hal Canary]

I was not thinking it through.