Redirecting stdout and stderr to a PyQt4 QTextEdit from a secondary thread

Question

Stack overflow. Once again, I come to you in a time of dire need, teetering precariously on the brink of insanity. This question - as may be evident from the title - is an amalgamation of several other questions I have seen answered here.

I have a PyQt application, and I want to re-route the stdout and stderr streams to a QTextEdit that is in my GUI without delay.

Initially, I found the following stack overflow answer: https://stackoverflow.com/a/17145093/629404

This works perfectly, but with one caveat: If stdout or stderr are updated multiple times while the CPU is processing a relatively longer method, all of the updates show up simultaneously when the main thread returns to the application loop. Unfortunately, I have a few methods which take up to 20 seconds to complete (networking related), and so the application becomes unresponsive - and the QTextEdit does not update - until they are finished.

In order to fix this problem, I delegated all of the GUI processing to the main thread, and I have been spawning off a second thread to handle the longer networking operations, using pyqtSignals to notify the main thread of when the work is finished and pass back results. Immediately when I began testing the code written this way, the python interpreter began crashing without any warning.

This is where it gets very frusterating: Python is crashing because - using the class from the included link above - I have assigned the sys.stdout/err streams to the QTextEdit widget; PyQt widgets cannot be modified from any thread other then the application thread, and since the updates to stdout and stderr are coming from the secondary worker thread that I created, they are violating this rule. I have commented out the section of code where I redirect the output streams, and sure enough, the program runs without error.

This brings me back to square one, and leaves me in a confusing situation; Assuming I continue to handle GUI related operations in the main thread and deal with computation and longer operations in a secondary thread (which I have come to understand is the best way to keep the application from blocking when the user triggers events), how can I redirect Stdout and Stderr from both threads to the QTextEdit widget? The class in the link above works just fine for the main thread, but kills python - for the reason described above - when updates come from the second thread.

Solution

Firstly, +1 for realising how thread-unsafe many of the examples on stack overflow are!

The solution is to use a thread-safe object (like a Python Queue.Queue) to mediate the transfer of information. I've attached some sample code below which redirects stdout to a Python Queue. This Queue is read by a QThread, which emits the contents to the main thread through Qt's signal/slot mechanism (emitting signals is thread-safe). The main thread then writes the text to a text edit.

Hope that is clear, feel free to ask questions if it is not!

EDIT: Note that the code example provided doesn't clean up QThreads nicely, so you'll get warnings printed when you quit. I'll leave it to you to extend to your use case and clean up the thread(s)

import sys
from Queue import Queue
from PyQt4.QtCore import *
from PyQt4.QtGui import *

# The new Stream Object which replaces the default stream associated with sys.stdout
# This object just puts data in a queue!
class WriteStream(object):
    def __init__(self,queue):
        self.queue = queue

    def write(self, text):
        self.queue.put(text)

# A QObject (to be run in a QThread) which sits waiting for data to come through a Queue.Queue().
# It blocks until data is available, and one it has got something from the queue, it sends
# it to the "MainThread" by emitting a Qt Signal 
class MyReceiver(QObject):
    mysignal = pyqtSignal(str)

    def __init__(self,queue,*args,**kwargs):
        QObject.__init__(self,*args,**kwargs)
        self.queue = queue

    @pyqtSlot()
    def run(self):
        while True:
            text = self.queue.get()
            self.mysignal.emit(text)

# An example QObject (to be run in a QThread) which outputs information with print
class LongRunningThing(QObject):
    @pyqtSlot()
    def run(self):
        for i in range(1000):
            print i

# An Example application QWidget containing the textedit to redirect stdout to
class MyApp(QWidget):
    def __init__(self,*args,**kwargs):
        QWidget.__init__(self,*args,**kwargs)

        self.layout = QVBoxLayout(self)
        self.textedit = QTextEdit()
        self.button = QPushButton('start long running thread')
        self.button.clicked.connect(self.start_thread)
        self.layout.addWidget(self.textedit)
        self.layout.addWidget(self.button)

    @pyqtSlot(str)
    def append_text(self,text):
        self.textedit.moveCursor(QTextCursor.End)
        self.textedit.insertPlainText( text )

    @pyqtSlot()
    def start_thread(self):
        self.thread = QThread()
        self.long_running_thing = LongRunningThing()
        self.long_running_thing.moveToThread(self.thread)
        self.thread.started.connect(self.long_running_thing.run)
        self.thread.start()

# Create Queue and redirect sys.stdout to this queue
queue = Queue()
sys.stdout = WriteStream(queue)

# Create QApplication and QWidget
qapp = QApplication(sys.argv)  
app = MyApp()
app.show()

# Create thread that will listen on the other end of the queue, and send the text to the textedit in our application
thread = QThread()
my_receiver = MyReceiver(queue)
my_receiver.mysignal.connect(app.append_text)
my_receiver.moveToThread(thread)
thread.started.connect(my_receiver.run)
thread.start()

qapp.exec_()