python, subprocess: reading output from subprocess

Question

I have following script:

#!/usr/bin/python

while True:
    x = raw_input()
    print x[::-1]

I am calling it from ipython:

In [5]: p = Popen('./script.py', stdin=PIPE)

In [6]: p.stdin.write('abc\n')
cba

and it works fine.

However, when I do this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.stdin.write('abc\n')

In [9]: p.stdout.read()

the interpreter hangs. What am I doing wrong? I would like to be able to both write and read from another process multiple times, to pass some tasks to this process. What do I need to do differently?

EDIT 1

If I use communicate, I get this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.communicate('abc\n')
Traceback (most recent call last):
  File "./script.py", line 4, in <module>
    x = raw_input()
EOFError: EOF when reading a line
Out[8]: ('cba\n', None)

EDIT 2

I tried flushing:

#!/usr/bin/python

import sys

while True:
        x = raw_input()
        print x[::-1]
        sys.stdout.flush()

and here:

In [5]: from subprocess import PIPE, Popen

In [6]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [7]: p.stdin.write('abc')

In [8]: p.stdin.flush()

In [9]: p.stdout.read()

but it hangs again.

Solution 1

I believe there are two problems at work here:

1) Your parent script calls p.stdout.read(), which will read all data until end-of-file. However, your child script runs in an infinite loop so end-of-file will never happen. Probably you want p.stdout.readline()?

2) In interactive mode, most programs do buffer only one line at a time. When run from another program, they buffer much more. The buffering improves efficiency in many cases, but causes problems when two programs need to communicate interactively.

After p.stdin.write('abc\n') add:

p.stdin.flush()

In your subprocess script, after print x[::-1] add the following within the loop:

sys.stdout.flush()

(and import sys at the top)

Solution 2

The subprocess method check_output can be useful for this:

output = subprocess.check_output('./script.py')

And output will be the stdout from the process. If you need stderr, too:

output = subprocess.check_output('./script.py', stderr=subprocess.STDOUT)

Because you avoid managing pipes directly, it may circumvent your issue.

Solution 3

If you'd like to pass several lines to script.py then you need to read/write simultaneously:

#!/usr/bin/env python
import sys
from subprocess import PIPE, Popen
from threading  import Thread

def print_output(out, ntrim=80):
    for line in out:
        print len(line)
        if len(line) > ntrim: # truncate long output
            line = line[:ntrim-2]+'..'
        print line.rstrip() 


if __name__=="__main__":
    p = Popen(['python', 'script.py'], stdin=PIPE, stdout=PIPE)
    Thread(target=print_output, args=(p.stdout,)).start()
    for s in ['abc', 'def', 'ab'*10**7, 'ghi']:
        print >>p.stdin, s
    p.stdin.close()
    sys.exit(p.wait()) #NOTE: read http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

Output:

4
cba
4
fed
20000001
bababababababababababababababababababababababababababababababababababababababa..
4
ihg

Where script.py:

#!/usr/bin/env python
"""Print reverse lines."""
while True:
    try: x = raw_input()
    except EOFError:
        break # no more input
    else:
        print x[::-1]

Or

#!/usr/bin/env python
"""Print reverse lines."""
import sys

for line in sys.stdin:
    print line.rstrip()[::-1]

Or

#!/usr/bin/env python
"""Print reverse lines."""
import fileinput

for line in fileinput.input(): # accept files specified as command line arguments
    print line.rstrip()[::-1]

Solution 4

You're probably tripping over Python's output buffering. Here's what python --help has to say about it.

-u     : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
         see man page for details on internal buffering relating to '-u'

Solution 5

When you are through writing to p.stdin, close it: p.stdin.close()

View more solutions

Comments

[gruszczy]

Can't I somehow force Python inside the code not to flush output? And why is it buffered, when it is pushed into pipe and not, when simply printed to the screen? Can I somehow flush the buffer programatically?

[ssokolow]

Output sent to pipes is buffered for performance reasons. (Flushing line-by-line is very inefficient, but you don't notice when the program spends most of its time waiting on the user) When flushing to pipes, buffering improves performance significantly, either improving completion times or reducing CPU load, depending on how I/O-bound the operation is.

[ssokolow]

As for flushing the buffer programmatically, you could try sys.stdout.flush() in the child process, but I've never used it and it's not really good architecture to use it in this manner. (It's more for use as part of a "sync to disk" process in concert with os.fsync) Better to let the parent process disable buffering as needed via either -u or the PYTHONUNBUFFERED environment variable.

[gruszczy]

This doesn't work for me. I have made an edit in my question.

[Daniel Stutzbach]

He wants to communicate multiple times with the subprocess. communicate can't do that.

[Daniel Stutzbach]

@gruszczy: Thanks for the update. I've updated my answer to address another problem.

[gruszczy]

Adding sys.stdout.flush() to the script and using p.stdout.readline finally helped. Thanks a lot for your help.

[Civilian]

What does the "print >>p.stdin, s" syntax do? I'm unfamiliar with ">>"

[jfs]

@Civilian: "print chevron" is similar to print(s, file=p.stdin) which is available after from __future__ import print_function.

[Benji Mizrahi]

Thank you Daniel Statzbash, for making my life easier. (flushing even after writing "\n")