How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out

Question

I want output from execute Test_Pipe.py, I tried following code on Linux but it did not work.

Test_Pipe.py

import time
while True :
    print "Someting ..."
    time.sleep(.1)

Caller.py

import subprocess as subp
import time

proc = subp.Popen(["python", "Test_Pipe.py"], stdout=subp.PIPE, stdin=subp.PIPE)

while True :
    data = proc.stdout.readline() #block / wait
    print data
    time.sleep(.1)

The line proc.stdout.readline() was blocked, so no data prints out.

Solution 1

You obviously can use subprocess.communicate but I think you are looking for real time input and output.

readline was blocked because the process is probably waiting on your input. You can read character by character to overcome this like the following:

import subprocess
import sys

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

while True:
    out = process.stdout.read(1)
    if out == '' and process.poll() != None:
        break
    if out != '':
        sys.stdout.write(out)
        sys.stdout.flush()

Solution 2

Nadia's snippet does work but calling read with a 1 byte buffer is highly unrecommended. The better way to do this would be to set the stdout file descriptor to nonblocking using fcntl

fcntl.fcntl(
    proc.stdout.fileno(),
    fcntl.F_SETFL,
    fcntl.fcntl(proc.stdout.fileno(), fcntl.F_GETFL) | os.O_NONBLOCK,
)

and then using select to test if the data is ready

while proc.poll() == None:
    readx = select.select([proc.stdout.fileno()], [], [])[0]
    if readx:
        chunk = proc.stdout.read()
        print chunk

She was correct in that your problem must be different from what you posted as Caller.py and Test_Pipe.py do work as provided.

• https://derrickpetzold.com/p/capturing-output-from-ffmpeg-python/

Solution 3

Test_Pipe.py buffers its stdout by default so proc in Caller.py doesn't see any output until the child's buffer is full (if the buffer size is 8KB then it takes around a minute to fill Test_Pipe.py's stdout buffer).

To make the output unbuffered (line-buffered for text streams) you could pass -u flag to the child Python script. It allows to read subprocess' output line by line in "real-time":

import sys
from subprocess import Popen, PIPE

proc = Popen([sys.executable, "-u", "Test_Pipe.py"], stdout=PIPE, bufsize=1)
for line in iter(proc.stdout.readline, b''):
    print line,
proc.communicate()

See links in Python: read streaming input from subprocess.communicate() on how to solve the block-buffering issue for non-Python child processes.

Solution 4

To avoid the many problems that can always arise with buffering for tasks such as "getting the subprocess's output to the main process in real time", I always recommend using pexpect for all non-Windows platform, wexpect on Windows, instead of subprocess, when such tasks are desired.

View more solutions

Comments

[jfs]

It won't produce the output any sooner than the code in the question due to the block-buffering issue. See my answer.

[jfs]

you don't need process.poll() in this case.

[Derrick Petzold]

Again you are using 1 byte buffer which is incredibly inefficient.

[Derrick Petzold]

The code does produce output in realtime. Did you have a problem with it?

[jfs]

follow the link in the comment.

[jfs]

@DerrickPetzold: wrong. bufsize=1 means "line-buffered". It uses the same buffer size as bufsize=-1. You could have found out that both of your comments are wrong if you were actually to run the code (compare time performance and measure the time before the first byte is read)

[sherrellbc]

What does it mean for data "to be ready?" Is it newline delineated or just if at least one byte of data is ready (even if the producer of such data is still writing to its stdout)?

[sherrellbc]

@me, J.F.Sebastian's answer below addresses this.

[pelos]

i like this way of doing it because you can print directly to screen and also store the results of the popen to do something with, and later on check error code etc... thumbs up =) i run it with out the bufsize and does work fine for me in my code.

[jfs]

to be clear: neither yours nor Nadia's answers produce the output in "real-time" due to the internal buffers in the child process. os.O_NONBLOCK in your code (in the parent process) has no effect on internal stdio buffers in the child process. For example, running your code produce 4K chunks instead of len(chunk) == 13 (one line). To see the difference, pass -u option (run [sys.executable or 'python2', '-u', 'Test_Pipe.py'] command).