Set Environmental Variables in Python with Popen

Question

I want to set an environmental variable in linux terminal through a python script. I seem to be able to set environmental variables when using os.environ['BLASTDB'] = '/path/to/directory' .

However I was initially trying to set this variable with subprocess.Popen with no success.

import subprocess
import shlex

cmd1 = 'export BLASTDB=/path/to/directory'
args = shlex.split(cmd1)
p = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()

Why does subprocess.Popen fail to set the environmental variable BLASTDB to '/path/to/directory'?

NOTE: This also fails when using:

import os
os.system('export BLASTDB=/path/to/directory')

Solution 1

Use the env parameter to set environment variables for a subprocess:

proc = subprocess.Popen(args, stdout=subprocess.PIPE,
                        env={'BLASTDB': '/path/to/directory'})

Per the docs:

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process’ environment, which is the default behavior.

Note: If specified, env must provide any variables required for the program to execute. On Windows, in order to run a side-by-side assembly the specified env must include a valid SystemRoot.

---

os.environ can used for accessing current environment variables of the python process. If your system also supports putenv, then os.environ can also be used for setting environment variables (and thus could be used instead of Popen's env parameter shown above). However, for some OSes such as FreeBSD and MacOS, setting os.environ may cause memory leaks, so setting os.environ is not a robust solution.

---

os.system('export BLASTDB=/path/to/directory') runs a subprocess which sets the BLASTDB environment variable only for that subprocess. Since that subprocess ends, it has no effect on subsequent subprocess.Popen calls.

Solution 2

As far as I know, you cannot really modify the executing process' environment from a subprocess or subshell, be it Python or bash itself. Environment variables are specific to the particular process you are on (at least on Unix, which you seem to be using).

Any child process spawned will usually inherit that environment, but only a copy of it. For instance, if you run bash from inside your terminal session and export a new environment variable, once you exit that subshell, your original shell will be untouched. Running Python is no different.

Comments

[Dologan]

I believe the OP is trying to set the environment variable for the executing terminal /through/ Python. Your answer will set the environment variable only for the subprocess itself.

[unutbu]

@Dologan: Even if that's true, the only purpose for setting the environment variable would be to affect subsequent subprocess calls, in which case using env is the answer.

[Admin]

@unutbu: From my testing env works great and temporarily sets the environmental variable BLASTDB for that specific subprocess, and everything moves along swimmingly. Thanks! For my purposes this is sufficient. os.environ is nice because it seems to set the environmental variable for the whole python subshell, and specifying env in each subprocess.Popen is not required.

[Admin]

So far I have not been able to modify the executing process' environment. For me, I only need temporary adjustment for specific subprocess commands. However, I was curious if this was possible. Thanks for the info.

[Bakuriu]

Actually, according to the documentation of os.environ, it can be used to modify the environment of the current process (if the OS provides a putenv syscall).

[unutbu]

@Bakuriu, on some OSes setting environ causes memory leaks, so it is not a reliable solution. But thank you for the comment; I've updated my answer to clarify this.

[Paul]

It is possible via the env keyword arg to subprocess.Popen as shown in the answer from unutbu

[Dologan]

The original question suggests that the OP wants to set the environment variable for the linux terminal itself, not just for a subprocess executed from within Python itself. In this case, my answer is still, to the best of my knowledge, valid and correct.

[alper]

Can we also add the currrent env variables along the one we are adding such as PATH? Because when I pass a binary as a comment, I had to provide its full path (which <binary_name>), if I pass env={} parameter @unutbu