Using python to run other programs
Solution 1
If you don't need to process the output in your code, only to show it to the user as it happens (it's not clear from your Q, and it seems that way from your own self-answer), simplest is:
rc = subprocess.call(
["cmd", "--thing", "foo", "--stuff", "bar",
"-a", "b", "input", "output"])
print "Return code was", rc
i.e., just avoid any use of pipes -- let stdout and stderr just show on the terminal. That should avoid any problem with buffering. Once you put pipes in the picture, buffering generally is a problem if you want to show output as it happens (I'm surprised your self-answer doesn't have that problem;-).
For both showing and capturing, BTW, I always recomment pexpect
(and wexpect
on Windows) exactly to work around the buffering issue.
Solution 2
You have to quote each field separately, ie. split the options from their arguments.
import subprocess
output = subprocess.call(["cmd", "--thing", "foo", "--stuff", "bar", "-a", "b", "input", "output"])
otherwise you are effectively running cmd like this
$ cmd --thing\ foo --stuff\ bar -a\ b input output
To get the output into a pipe you need to call it slightly differently
import subprocess
output = subprocess.Popen(["cmd", "--thing", "foo", "--stuff", "bar", "-a", "b", "input", "output"],stdout=subprocess.PIPE)
output.stdout # <open file '<fdopen>', mode 'rb'>
Solution 3
Wouldn't commands.getstatusoutput() work? It'll return your status right away pretty sure.
Solution 4
A coworker just showed me this:
import os
import sys
for line in os.popen("cmd --thing foo --stuff bar -a b input output", "r"):
print line
sys.stdout.flush()
and it is working :)
Related videos on Youtube
Paul Tarjan
I'm a Distinguished Engineer at Robinhood. I used to be the Tech Lead of Developer Productivity at Stripe where I built Sorbet. Before that I was the CTO and cofounder at Trimian. Before that I was a Software Engineer at Facebook on HHVM and the Open Graph. Before that I was the Tech Lead for Yahoo! SearchMonkey. See my homepage for more.
Updated on April 16, 2022Comments
-
Paul Tarjan about 2 years
I have a command that works great on the command line. It has lots of arguments like
cmd --thing foo --stuff bar -a b input output
I want to run this from python and block waiting for it to complete. As the script prints things to
stdout
andstderr
I want it to be immediately shown to the user.What is the right module for this?
I've tried:
import commands output = commands.getoutput("cmd --thing foo --stuff bar -a b input output") print output
this works great except the
stdout
isn't returned until the end.
import os os.system("cmd --thing foo --stuff bar -a b input output")
this prints all the output when the cmd is actually finished.
import subprocess subprocess.call(["cmd", "--thing foo", "--stuff bar", "-a b", "input", "output"])
this doesn't pass the parameters correctly somehow (I haven't been able to find the exact problem, but
cmd
is rejecting my input). If I putecho
as the first parameter, it prints out the command which works perfectly when I paste it directly into the terminal.
import subprocess subprocess.call("cmd --thing foo --stuff bar -a b input output")
exactly the same as above.
-
John La Rooy about 14 yearsYou should use
subprocess
in preference topopen
. For one thing it saves you worrying about escaping parameters -
jathanism about 14 yearsIt does, but be warned that it is POSIX only (no Windows).
-
Paul Tarjan about 14 years
subprocess
doesn't seem to be printing the output as it is returned. Can I make that happen? -
user1066101 about 14 years@Paul Tarjan: That's a duplicate question. Please search for
"[python] subprocess"
and you'll get dozens of answers to that question. -
Oben Sonne about 14 yearsAnother reason: Since Python 2.6
os.popen()
is classified as deprecated in favor ofsubprocess
(see docs.python.org/library/os.html#os.popen).