Read console output of another program in Python

Question

There are two programs - parent.py and myscript.py, shown below. parent.py keeps printing messages in the console. And myscript.py needs access to what parent.py prints.

parent.py:

import time

while 1:
     time.sleep(1)
     print "test"

myscript.py:

import subprocess
p = None

p = subprocess.Popen(['python', 'parent.py'],
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT,
                 )
for line in p.stdout:
       print line

I want myscript.py to catch the output of parent.py dynamically. As it can be seen that parent.py has an infinite loop - the script is never gonna stop, but whenever it outputs the word 'test', myscript.py should also output the same. But myscript.py doesn't give out any output.

I tried replacing the code in parent.py with just

print "Hello"

Since the program finished executing, myscript.py also printed "Hello". So I guess unless the execution of parent.py is complete, myscript.py doesn't read it. Is that whats happening?

I also tried replacing the loop in myscript.py with this snippet:

for line in iter(p.stdout.readline, ''):
    print line

Still, no output.

My question is: How do I get myscript.py dynamically read all that is being printed by parent.py even when parent.py is never going to complete its execution.

(I found similar questions, but they either didn't work or there were no answers at all in some of them.)

Solution

It is a block buffering issue: when stdout is redirected to a pipe parent.py accumulates its output in an internal buffer (~4K-8K) i.e., you won't see anything for around an hour until the buffer overflows.

To disable buffering, pass -u command-line option:

#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE, STDOUT

script_path = os.path.join(get_script_dir(), 'parent.py')
p = Popen([sys.executable, '-u', script_path],
          stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print line,
p.wait()

See:

• Python: read streaming input from subprocess.communicate()
• Call python script with input with in a python script using subprocess

Note: if you don't need to do anything with the output then just drop stdout=PIPE, to see the output in the console:

subprocess.check_call([sys.executable, script_path])

Comments

[Krish]

Works! Also, a quick question (not sure if its a quick workaround or it involves more elaborate approach): is it possible to achieve the same with a running program? I mean, in this case - myscript.py executes parent.py and then returns the output to myscript. I'm looking to extract console output of code thats already running and extracting its output. Like identifying the process > getting process id > get output from console. Should I ask this as a separate question or can I tweak this script itself?

[jfs]

@Krish: "already running" case (especially if you can't control how the process is started) is a separate question: make sure to provide the context and motivation: what problem are you trying to solve and why do you think intercepting the output will help.