Advantages of subprocess over os.system

Question

I have recently came across a few posts on stack overflow saying that subprocess is much better than os.system, however I am having difficulty finding the exact advantages.

Some examples of things I have run into: https://docs.python.org/3/library/os.html#os.system

"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."

No idea in what ways it is more powerful though, I know it is easier in many ways to use subprocess but is it actually more powerful in some way?

Another example is:

https://stackoverflow.com/a/89243/3339122

The advantage of subprocess vs system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

This post which has 2600+ votes. Again could not find any elaboration on what was meant by better error handling or real status code.

Top comment on that post is:

Can't see why you'd use os.system even for quick/dirty/one-time. subprocess seems so much better.

Again, I understand it makes some things slightly easier, but I hardly can understand why for example:

subprocess.call("netsh interface set interface \"Wi-Fi\" enable", shell=True)

is any better than

os.system("netsh interface set interface \"Wi-Fi\" enabled")

Can anyone explain some reasons it is so much better?

Solution 1

First of all, you are cutting out the middleman; subprocess.call by default avoids spawning a shell that examines your command, and directly spawns the requested process. This is important because, besides the efficiency side of the matter, you don't have much control over the default shell behavior, and it actually typically works against you regarding escaping.

In particular, do not do this:

subprocess.call('netsh interface set interface "Wi-Fi" enable')

since

If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.

Instead, you'll do:

subprocess.call(["netsh", "interface", "set", "interface", "Wi-Fi", "enable"])

Notice that here all the escaping nightmares are gone. subprocess handles escaping (if the OS wants arguments as a single string - such as Windows) or passes the separated arguments straight to the relevant syscall (execvp on UNIX).

Compare this with having to handle the escaping yourself, especially in a cross-platform way (cmd doesn't escape in the same way as POSIX sh), especially with the shell in the middle messing with your stuff (trust me, you don't want to know what unholy mess is to provide a 100% safe escaping for your command when calling cmd /k).

Also, when using subprocess without the shell in the middle you are sure you are getting correct return codes. If there's a failure launching the process you get a Python exception, if you get a return code it's actually the return code of the launched program. With os.system you have no way to know if the return code you get comes from the launched command (which is generally the default behavior if the shell manages to launch it) or it is some error from the shell (if it didn't manage to launch it).

---

Besides arguments splitting/escaping and return code, you have way better control over the launched process. Even with subprocess.call (which is the most basic utility function over subprocess functionalities) you can redirect stdin, stdout and stderr, possibly communicating with the launched process. check_call is similar and it avoids the risk of ignoring a failure exit code. check_output covers the common use case of check_call + capturing all the program output into a string variable.

Once you get past call & friends (which is blocking just as os.system), there are way more powerful functionalities - in particular, the Popen object allows you to work with the launched process asynchronously. You can start it, possibly talk with it through the redirected streams, check if it is running from time to time while doing other stuff, waiting for it to complete, sending signals to it and killing it - all stuff that is way besides the mere synchronous "start process with default stdin/stdout/stderr through the shell and wait it to finish" that os.system provides.

---

So, to sum it up, with subprocess:

• even at the most basic level (call & friends), you:
  • avoid escaping problems by passing a Python list of arguments;
  • avoid the shell messing with your command line;
  • either you have an exception or the true exit code of the process you launched; no confusion about program/shell exit code;
  • have the possibility to capture stdout and in general redirect the standard streams;
• when you use Popen:
  • you aren't restricted to a synchronous interface, but you can actually do other stuff while the subprocess run;
  • you can control the subprocess (check if it is running, communicate with it, kill it).

Given that subprocess does way more than os.system can do - and in a safer, more flexible (if you need it) way - there's just no reason to use system instead.

Solution 2

There are many reasons, but the main reason is mentioned directly in the docstring:

>>> os.system.__doc__
'Execute the command in a subshell.'

For almost all cases where you need a subprocess, it is undesirable to spawn a subshell. This is unnecessary and wasteful, it adds an extra layer of complexity, and introduces several new vulnerabilities and failure modes. Using subprocess module cuts out the middleman.

Comments

[Lain]

@JonathonReinhart Just for the record. It actually works fine with or without the shell=True. I have indeed read the documentation. It is also not the question at hand of why my code is or isn't working. I can update the example if you would like, question is still the differences between them. See: stackoverflow.com/questions/3172470/… before you jump to conclusions or do other research. I know how the code works, I just don't know why I am using it over something else.

[Lain]

Thanks, I'll thumbs up the comment. Doesn't exactly answer the better error handling, real status codes, or other things I had mentioned in my post people talk about. Even your post does the same as theirs, unsure what new vulnerabilities/failure modes you mean, and alos unsure which casses it would be desirable to spawn a subshell. Can't imagine the small wastefulness from this would gather so much support is all.

[wim]

A shell allows you to interact with the operating system. Python also has all the power necessary to interact with the operating system. Why ask Python to ask the shell to interact with the operating system? The extra complexity comes from having to translate things like pipes and escapes into the shell language. To learn about the vulnerabilities, search for shell injection.