Interactive input/output using Python

Question

I have a program that interacts with the user (acts like a shell), and I want to run it using the Python subprocess module interactively. That means, I want the possibility to write to standard input and immediately get the output from standard output. I tried many solutions offered here, but none of them seems to work for my needs.

The code I've written is based on Running an interactive command from within Python.

import Queue
import threading
import subprocess
def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

def getOutput(outQueue):
    outStr = ''
    try:
        while True: # Adds output from the queue until it is empty
            outStr += outQueue.get_nowait()

    except Queue.Empty:
        return outStr

p = subprocess.Popen("./a.out", stdin=subprocess.PIPE, stout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize = 1)
#p = subprocess.Popen("./a.out", stdin=subprocess.PIPE, stout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, universal_newlines=True)

outQueue = Queue()
errQueue = Queue()

outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = Thread(target=enqueue_output, args=(p.stderr, errQueue))

outThread.daemon = True
errThread.daemon = True

outThread.start()
errThread.start()

p.stdin.write("1\n")
p.stdin.flush()
errors = getOutput(errQueue)
output = getOutput(outQueue)

p.stdin.write("5\n")
p.stdin.flush()
erros = getOutput(errQueue)
output = getOutput(outQueue)

The problem is that the queue remains empty, as if there is no output. Only if I write to standard input all the input that the program needs to execute and terminate, then I get the output (which is not what I want). For example, if I do something like:

p.stdin.write("1\n5\n")
errors = getOutput(errQueue)
output = getOutput(outQueue)

Is there a way to do what I want to do?

---

The script will run on a Linux machine. I changed my script and deleted the universal_newlines=True + set the bufsize to 1 and flushed standard input immediately after write. Still I don't get any output.

Second try:

I tried this solution and it works for me:

from subprocess import Popen, PIPE

fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1\n")
out = fr.read()
p.stdin.write("5\n")
out = fr.read()
fw.close()
fr.close()

Solution 1

Two solutions for this issue on Linux:

First one is to use a file to write the output to, and read from it simultaneously:

from subprocess import Popen, PIPE

fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1\n")
out = fr.read()
p.stdin.write("5\n")
out = fr.read()
fw.close()
fr.close()

Second, as J.F. Sebastian offered, is to make p.stdout and p.stderr pipes non-blocking using fnctl module:

import os
import fcntl
from subprocess import Popen, PIPE  
def setNonBlocking(fd):
    """
    Set the file description of the given file descriptor to non-blocking.
    """
    flags = fcntl.fcntl(fd, fcntl.F_GETFL)
    flags = flags | os.O_NONBLOCK
    fcntl.fcntl(fd, fcntl.F_SETFL, flags)

p = Popen("./a.out", stdin = PIPE, stdout = PIPE, stderr = PIPE, bufsize = 1)
setNonBlocking(p.stdout)
setNonBlocking(p.stderr)

p.stdin.write("1\n")
while True:
    try:
        out1 = p.stdout.read()
    except IOError:
        continue
    else:
        break
out1 = p.stdout.read()
p.stdin.write("5\n")
while True:
    try:
        out2 = p.stdout.read()
    except IOError:
        continue
    else:
        break

Solution 2

None of the current answers worked for me. At the end, I've got this working:

import subprocess


def start(executable_file):
    return subprocess.Popen(
        executable_file,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)


def read(process):
    return process.stdout.readline().decode("utf-8").strip()


def write(process, message):
    process.stdin.write(f"{message.strip()}\n".encode("utf-8"))
    process.stdin.flush()


def terminate(process):
    process.stdin.close()
    process.terminate()
    process.wait(timeout=0.2)


process = start("./dummy.py")
write(process, "hello dummy")
print(read(process))
terminate(process)

Tested with this dummy.py script:

#!/usr/bin/env python3.6

import random
import time

while True:
    message = input()
    time.sleep(random.uniform(0.1, 1.0)) # simulates process time
    print(message[::-1])

The caveats are (all managed in the functions):

• Input/output always lines with newline.
• Flush child's stdin after every write.
• Use readline() from child's stdout.

It's a pretty simple solution in my opinion (not mine, I found it here: https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/). I was using Python 3.6.

Solution 3

Here is an interactive shell. You have to run read() on a separate thread, otherwise it will block the write()

import sys
import os
import subprocess
from subprocess import Popen, PIPE
import threading


class LocalShell(object):
    def __init__(self):
        pass

    def run(self):
        env = os.environ.copy()
        p = Popen('/bin/bash', stdin=PIPE, stdout=PIPE, stderr=subprocess.STDOUT, shell=True, env=env)
        sys.stdout.write("Started Local Terminal...\r\n\r\n")

        def writeall(p):
            while True:
                # print("read data: ")
                data = p.stdout.read(1).decode("utf-8")
                if not data:
                    break
                sys.stdout.write(data)
                sys.stdout.flush()

        writer = threading.Thread(target=writeall, args=(p,))
        writer.start()

        try:
            while True:
                d = sys.stdin.read(1)
                if not d:
                    break
                self._write(p, d.encode())

        except EOFError:
            pass

    def _write(self, process, message):
        process.stdin.write(message)
        process.stdin.flush()


shell = LocalShell()
shell.run()

Comments

[user202729]

related.

[Charlie Parker]

ok the code doesn't show in the bounty, so I did a gist for it gist.github.com/brando90/99b10cdc73dc6b604ca661712c1c7b0d I am just trying to test it with a real python process in the background.

[Charlie Parker]

did you take a look at this: docs.python.org/3/library/asyncio.html ?

[user240515]

related: stackoverflow.com/q/375427/240515

[user240515]

If anyone wants to do this in modern-day Python, I've posted a definitive answer here: stackoverflow.com/a/56051270/240515

[Talor Abramovich]

I don't exactly understand what you mean. What I wanted to do at first is to read only partial output (one line)

[Talor Abramovich]

I didn't understand your example. What does this line: p = Popen([sys.executable, "-u", '-c' 'for line in iter(input, ""): print("a"*int(line)*10**6)'], stdin=PIPE, stdout=PIPE, bufsize=1) mean?

[Talor Abramovich]

You're right, I've edited my second solution as I used it. The problem was that when I tried the solution at first, I've tried it on the python interpreter (I haven't written the script, only tested it manually) so it worked (due to human slow response time). When I tried to write the script, I've encountered the problem you've mentioned so I've added the while loop, until the input is ready. I actually get the whole output or nothing at all (IOError exception), will it be helpful if I upload the a.out source code (it's a C program)?

[Pod]

Unfortunately the script I want to communicate with uses stty -echo which causes it to complain of 'standard input': Inappropriate ioctl for device when trying to use the first method. (The second method never seems to read the output properly.. assuming the script is outputting)

[Dmitry]

The second solution hangs forever, when calling a simple C program requesting 2 inputs with scanf("%s", buf)

[Admin]

I tried your first solution, and stuck on "./a.out" that line. I ran this code on Ubuntu 16 with Python 3.5, getting this error, any response would be appreciated. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.5/subprocess.py", line 947, in init restore_signals, start_new_session) File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child raise child_exception_type(errno_num, err_msg) OSError: [Errno 8] Exec format error

[Charlie Parker]

I am having issues. What if you have a Popen that isn't simply a file that one reads at but a process that needs actual commands. e.g. p = subprocess.Popen(['python'],stdin=subprocess.PIPE,stdout=frw‌​,stderr=frw,). I am having difficulties actually sending commands to my dummy (python) process and reading its output on the fly. How does one do this?

[Charlie Parker]

is there any reason why docs.python.org/3/library/asyncio.html wasn't mentioned? Is it not useful for this task?

[Chris du Plessis]

Thank you! I made it work for Windows by changing 2 lines: p = Popen('/bin/bash', stdin=PIPE... to p = Popen(['cmd.exe'], stdin=PIPE... and line data = p.stdout.read(1).decode("utf-8") to data = p.stdout.read(1).decode("cp437"). Just if anyone wants to run this on Windows instead.

[USERNAME GOES HERE]

Your answer is the best! Thank you!

[Ben]

Just want to comment saying I've been trying to do this seemingly very simple task for about 45 minutes now, and stdin.flush turned out to be the solution to everything. I haven't seen this line included in any other answer I've read anywhere.

[mrkbutty]

As well as modifying from bash to cmd you can also add the argument "text=True" to the end of the Popen and then the encode/decodes can also be removed. Great answer!