Reading/writing to a Popen() subprocess

Question

I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.

This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()

p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()

Solution 1

I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:

p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
                     stdout=subprocess.PIPE,
                     stdin=subprocess.PIPE)

and your code should work fine

Solution 2

sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.

Try this:

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()

print "Reading result 1:"
print p.stdout.readline()

print "Reading result 2:"
print p.stdout.readline()

Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().

Comments

[Mats Ekberg]

But sed does not act that way when running the command in the shell directly. If you do that, the response comes after each line. And the problem remains even if I call p.stdin.flush() after writing. Also, in real life, I have written the called program as well, there is no buffering and it behaves in the same way. I'm not convinced that buffering is the problem here.