Background thread with QThread in PyQt

Question

I have a program which interfaces with a radio I am using via a gui I wrote in PyQt. Obviously one of the main functions of the radio is to transmit data, but to do this continuously, I have to loop the writes, which causes the gui to hang. Since I have never dealt with threading, I tried to get rid of these hangs using QCoreApplication.processEvents(). The radio needs to sleep between transmissions, though, so the gui still hangs based on how long these sleeps last.

Is there a simple way to fix this using QThread? I have looked for tutorials on how to implement multithreading with PyQt, but most of them deal with setting up servers and are much more advanced than I need them to be. I honestly don't even really need my thread to update anything while it is running, I just need to start it, have it transmit in the background, and stop it.

Solution 1

I created a little example that shows 3 different and simple ways of dealing with threads. I hope it will help you find the right approach to your problem.

import sys
import time

from PyQt5.QtCore import (QCoreApplication, QObject, QRunnable, QThread,
                          QThreadPool, pyqtSignal)


# Subclassing QThread
# http://qt-project.org/doc/latest/qthread.html
class AThread(QThread):

    def run(self):
        count = 0
        while count < 5:
            time.sleep(1)
            print("A Increasing")
            count += 1

# Subclassing QObject and using moveToThread
# http://blog.qt.digia.com/blog/2007/07/05/qthreads-no-longer-abstract
class SomeObject(QObject):

    finished = pyqtSignal()

    def long_running(self):
        count = 0
        while count < 5:
            time.sleep(1)
            print("B Increasing")
            count += 1
        self.finished.emit()

# Using a QRunnable
# http://qt-project.org/doc/latest/qthreadpool.html
# Note that a QRunnable isn't a subclass of QObject and therefore does
# not provide signals and slots.
class Runnable(QRunnable):

    def run(self):
        count = 0
        app = QCoreApplication.instance()
        while count < 5:
            print("C Increasing")
            time.sleep(1)
            count += 1
        app.quit()


def using_q_thread():
    app = QCoreApplication([])
    thread = AThread()
    thread.finished.connect(app.exit)
    thread.start()
    sys.exit(app.exec_())

def using_move_to_thread():
    app = QCoreApplication([])
    objThread = QThread()
    obj = SomeObject()
    obj.moveToThread(objThread)
    obj.finished.connect(objThread.quit)
    objThread.started.connect(obj.long_running)
    objThread.finished.connect(app.exit)
    objThread.start()
    sys.exit(app.exec_())

def using_q_runnable():
    app = QCoreApplication([])
    runnable = Runnable()
    QThreadPool.globalInstance().start(runnable)
    sys.exit(app.exec_())

if __name__ == "__main__":
    #using_q_thread()
    #using_move_to_thread()
    using_q_runnable()

Solution 2

Take this answer updated for PyQt5, python 3.4

Use this as a pattern to start a worker that does not take data and return data as they are available to the form.

1 - Worker class is made smaller and put in its own file worker.py for easy memorization and independent software reuse.

2 - The main.py file is the file that defines the GUI Form class

3 - The thread object is not subclassed.

4 - Both thread object and the worker object belong to the Form object

5 - Steps of the procedure are within the comments.

# worker.py
from PyQt5.QtCore import QThread, QObject, pyqtSignal, pyqtSlot
import time


class Worker(QObject):
    finished = pyqtSignal()
    intReady = pyqtSignal(int)


    @pyqtSlot()
    def procCounter(self): # A slot takes no params
        for i in range(1, 100):
            time.sleep(1)
            self.intReady.emit(i)

        self.finished.emit()

And the main file is:

  # main.py
  from PyQt5.QtCore import QThread
  from PyQt5.QtWidgets import QApplication, QLabel, QWidget, QGridLayout
  import sys
  import worker


  class Form(QWidget):

    def __init__(self):
       super().__init__()
       self.label = QLabel("0")

       # 1 - create Worker and Thread inside the Form
       self.obj = worker.Worker()  # no parent!
       self.thread = QThread()  # no parent!

       # 2 - Connect Worker`s Signals to Form method slots to post data.
       self.obj.intReady.connect(self.onIntReady)

       # 3 - Move the Worker object to the Thread object
       self.obj.moveToThread(self.thread)

       # 4 - Connect Worker Signals to the Thread slots
       self.obj.finished.connect(self.thread.quit)

       # 5 - Connect Thread started signal to Worker operational slot method
       self.thread.started.connect(self.obj.procCounter)

       # * - Thread finished signal will close the app if you want!
       #self.thread.finished.connect(app.exit)

       # 6 - Start the thread
       self.thread.start()

       # 7 - Start the form
       self.initUI()


    def initUI(self):
        grid = QGridLayout()
        self.setLayout(grid)
        grid.addWidget(self.label,0,0)

        self.move(300, 150)
        self.setWindowTitle('thread test')
        self.show()

    def onIntReady(self, i):
        self.label.setText("{}".format(i))
        #print(i)

    app = QApplication(sys.argv)

    form = Form()

    sys.exit(app.exec_())

Solution 3

According to the Qt developers, subclassing QThread is incorrect (see http://blog.qt.io/blog/2010/06/17/youre-doing-it-wrong/). But that article is really hard to understand (plus the title is a bit condescending). I found a better blog post that gives a more detailed explanation about why you should use one style of threading over another: http://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/

In my opinion, you should probably never subclass thread with the intent to overload the run method. While that does work, you're basically circumventing how Qt wants you to work. Plus you'll miss out on things like events and proper thread safe signals and slots. Plus as you'll likely see in the above blog post, the "correct" way of threading forces you to write more testable code.

Here's a couple of examples of how to take advantage of QThreads in PyQt (I posted a separate answer below that properly uses QRunnable and incorporates signals/slots, that answer is better if you have a lot of async tasks that you need to load balance).

import sys
from PyQt4 import QtCore
from PyQt4 import QtGui
from PyQt4.QtCore import Qt

# very testable class (hint: you can use mock.Mock for the signals)
class Worker(QtCore.QObject):
    finished = QtCore.pyqtSignal()
    dataReady = QtCore.pyqtSignal(list, dict)

    @QtCore.pyqtSlot()
    def processA(self):
        print "Worker.processA()"
        self.finished.emit()

    @QtCore.pyqtSlot(str, list, list)
    def processB(self, foo, bar=None, baz=None):
        print "Worker.processB()"
        for thing in bar:
            # lots of processing...
            self.dataReady.emit(['dummy', 'data'], {'dummy': ['data']})
        self.finished.emit()


class Thread(QtCore.QThread):
    """Need for PyQt4 <= 4.6 only"""
    def __init__(self, parent=None):
        QtCore.QThread.__init__(self, parent)

     # this class is solely needed for these two methods, there
     # appears to be a bug in PyQt 4.6 that requires you to
     # explicitly call run and start from the subclass in order
     # to get the thread to actually start an event loop

    def start(self):
        QtCore.QThread.start(self)

    def run(self):
        QtCore.QThread.run(self)


app = QtGui.QApplication(sys.argv)

thread = Thread() # no parent!
obj = Worker() # no parent!
obj.moveToThread(thread)

# if you want the thread to stop after the worker is done
# you can always call thread.start() again later
obj.finished.connect(thread.quit)

# one way to do it is to start processing as soon as the thread starts
# this is okay in some cases... but makes it harder to send data to
# the worker object from the main gui thread.  As you can see I'm calling
# processA() which takes no arguments
thread.started.connect(obj.processA)
thread.start()

# another way to do it, which is a bit fancier, allows you to talk back and
# forth with the object in a thread safe way by communicating through signals
# and slots (now that the thread is running I can start calling methods on
# the worker object)
QtCore.QMetaObject.invokeMethod(obj, 'processB', Qt.QueuedConnection,
                                QtCore.Q_ARG(str, "Hello World!"),
                                QtCore.Q_ARG(list, ["args", 0, 1]),
                                QtCore.Q_ARG(list, []))

# that looks a bit scary, but its a totally ok thing to do in Qt,
# we're simply using the system that Signals and Slots are built on top of,
# the QMetaObject, to make it act like we safely emitted a signal for 
# the worker thread to pick up when its event loop resumes (so if its doing
# a bunch of work you can call this method 10 times and it will just queue
# up the calls.  Note: PyQt > 4.6 will not allow you to pass in a None
# instead of an empty list, it has stricter type checking

app.exec_()

# Without this you may get weird QThread messages in the shell on exit
app.deleteLater()        

Solution 4

Very nice example from Matt, I fixed the typo and also pyqt4.8 is common now so I removed the dummy class as well and added an example for the dataReady signal

# -*- coding: utf-8 -*-
import sys
from PyQt4 import QtCore, QtGui
from PyQt4.QtCore import Qt


# very testable class (hint: you can use mock.Mock for the signals)
class Worker(QtCore.QObject):
    finished = QtCore.pyqtSignal()
    dataReady = QtCore.pyqtSignal(list, dict)

    @QtCore.pyqtSlot()
    def processA(self):
        print "Worker.processA()"
        self.finished.emit()

    @QtCore.pyqtSlot(str, list, list)
    def processB(self, foo, bar=None, baz=None):
        print "Worker.processB()"
        for thing in bar:
            # lots of processing...
            self.dataReady.emit(['dummy', 'data'], {'dummy': ['data']})
        self.finished.emit()


def onDataReady(aList, aDict):
    print 'onDataReady'
    print repr(aList)
    print repr(aDict)


app = QtGui.QApplication(sys.argv)

thread = QtCore.QThread()  # no parent!
obj = Worker()  # no parent!
obj.dataReady.connect(onDataReady)

obj.moveToThread(thread)

# if you want the thread to stop after the worker is done
# you can always call thread.start() again later
obj.finished.connect(thread.quit)

# one way to do it is to start processing as soon as the thread starts
# this is okay in some cases... but makes it harder to send data to
# the worker object from the main gui thread.  As you can see I'm calling
# processA() which takes no arguments
thread.started.connect(obj.processA)
thread.finished.connect(app.exit)

thread.start()

# another way to do it, which is a bit fancier, allows you to talk back and
# forth with the object in a thread safe way by communicating through signals
# and slots (now that the thread is running I can start calling methods on
# the worker object)
QtCore.QMetaObject.invokeMethod(obj, 'processB', Qt.QueuedConnection,
                                QtCore.Q_ARG(str, "Hello World!"),
                                QtCore.Q_ARG(list, ["args", 0, 1]),
                                QtCore.Q_ARG(list, []))

# that looks a bit scary, but its a totally ok thing to do in Qt,
# we're simply using the system that Signals and Slots are built on top of,
# the QMetaObject, to make it act like we safely emitted a signal for
# the worker thread to pick up when its event loop resumes (so if its doing
# a bunch of work you can call this method 10 times and it will just queue
# up the calls.  Note: PyQt > 4.6 will not allow you to pass in a None
# instead of an empty list, it has stricter type checking

app.exec_()

Solution 5

In PyQt there are a lot of options for getting asynchronous behavior. For things that need event processing (ie. QtNetwork, etc) you should use the QThread example I provided in my other answer on this thread. But for the vast majority of your threading needs, I think this solution is far superior than the other methods.

The advantage of this is that the QThreadPool schedules your QRunnable instances as tasks. This is similar to the task pattern used in Intel's TBB. It's not quite as elegant as I like but it does pull off excellent asynchronous behavior.

This allows you to utilize most of the threading power of Qt in Python via QRunnable and still take advantage of signals and slots. I use this same code in several applications, some that make hundreds of asynchronous REST calls, some that open files or list directories, and the best part is using this method, Qt task balances the system resources for me.

import time
from PyQt4 import QtCore
from PyQt4 import QtGui
from PyQt4.QtCore import Qt


def async(method, args, uid, readycb, errorcb=None):
    """
    Asynchronously runs a task

    :param func method: the method to run in a thread
    :param object uid: a unique identifier for this task (used for verification)
    :param slot updatecb: the callback when data is receieved cb(uid, data)
    :param slot errorcb: the callback when there is an error cb(uid, errmsg)

    The uid option is useful when the calling code makes multiple async calls
    and the callbacks need some context about what was sent to the async method.
    For example, if you use this method to thread a long running database call
    and the user decides they want to cancel it and start a different one, the
    first one may complete before you have a chance to cancel the task.  In that
    case, the "readycb" will be called with the cancelled task's data.  The uid
    can be used to differentiate those two calls (ie. using the sql query).

    :returns: Request instance
    """
    request = Request(method, args, uid, readycb, errorcb)
    QtCore.QThreadPool.globalInstance().start(request)
    return request


class Request(QtCore.QRunnable):
    """
    A Qt object that represents an asynchronous task

    :param func method: the method to call
    :param list args: list of arguments to pass to method
    :param object uid: a unique identifier (used for verification)
    :param slot readycb: the callback used when data is receieved
    :param slot errorcb: the callback used when there is an error

    The uid param is sent to your error and update callbacks as the
    first argument. It's there to verify the data you're returning

    After created it should be used by invoking:

    .. code-block:: python

       task = Request(...)
       QtCore.QThreadPool.globalInstance().start(task)

    """
    INSTANCES = []
    FINISHED = []
    def __init__(self, method, args, uid, readycb, errorcb=None):
        super(Request, self).__init__()
        self.setAutoDelete(True)
        self.cancelled = False

        self.method = method
        self.args = args
        self.uid = uid
        self.dataReady = readycb
        self.dataError = errorcb

        Request.INSTANCES.append(self)

        # release all of the finished tasks
        Request.FINISHED = []

    def run(self):
        """
        Method automatically called by Qt when the runnable is ready to run.
        This will run in a separate thread.
        """
        # this allows us to "cancel" queued tasks if needed, should be done
        # on shutdown to prevent the app from hanging
        if self.cancelled:
            self.cleanup()
            return

        # runs in a separate thread, for proper async signal/slot behavior
        # the object that emits the signals must be created in this thread.
        # Its not possible to run grabber.moveToThread(QThread.currentThread())
        # so to get this QObject to properly exhibit asynchronous
        # signal and slot behavior it needs to live in the thread that
        # we're running in, creating the object from within this thread
        # is an easy way to do that.
        grabber = Requester()
        grabber.Loaded.connect(self.dataReady, Qt.QueuedConnection)
        if self.dataError is not None:
            grabber.Error.connect(self.dataError, Qt.QueuedConnection)

        try:
            result = self.method(*self.args)
            if self.cancelled:
                # cleanup happens in 'finally' statement
                return
            grabber.Loaded.emit(self.uid, result)
        except Exception as error:
            if self.cancelled:
                # cleanup happens in 'finally' statement
                return
            grabber.Error.emit(self.uid, unicode(error))
        finally:
            # this will run even if one of the above return statements
            # is executed inside of the try/except statement see:
            # https://docs.python.org/2.7/tutorial/errors.html#defining-clean-up-actions
            self.cleanup(grabber)

    def cleanup(self, grabber=None):
        # remove references to any object or method for proper ref counting
        self.method = None
        self.args = None
        self.uid = None
        self.dataReady = None
        self.dataError = None

        if grabber is not None:
            grabber.deleteLater()

        # make sure this python obj gets cleaned up
        self.remove()

    def remove(self):
        try:
            Request.INSTANCES.remove(self)

            # when the next request is created, it will clean this one up
            # this will help us avoid this object being cleaned up
            # when it's still being used
            Request.FINISHED.append(self)
        except ValueError:
            # there might be a race condition on shutdown, when shutdown()
            # is called while the thread is still running and the instance
            # has already been removed from the list
            return

    @staticmethod
    def shutdown():
        for inst in Request.INSTANCES:
            inst.cancelled = True
        Request.INSTANCES = []
        Request.FINISHED = []


class Requester(QtCore.QObject):
    """
    A simple object designed to be used in a separate thread to allow
    for asynchronous data fetching
    """

    #
    # Signals
    #

    Error = QtCore.pyqtSignal(object, unicode)
    """
    Emitted if the fetch fails for any reason

    :param unicode uid: an id to identify this request
    :param unicode error: the error message
    """

    Loaded = QtCore.pyqtSignal(object, object)
    """
    Emitted whenever data comes back successfully

    :param unicode uid: an id to identify this request
    :param list data: the json list returned from the GET
    """

    NetworkConnectionError = QtCore.pyqtSignal(unicode)
    """
    Emitted when the task fails due to a network connection error

    :param unicode message: network connection error message
    """

    def __init__(self, parent=None):
        super(Requester, self).__init__(parent)


class ExampleObject(QtCore.QObject):
    def __init__(self, parent=None):
        super(ExampleObject, self).__init__(parent)
        self.uid = 0
        self.request = None

    def ready_callback(self, uid, result):
        if uid != self.uid:
            return
        print "Data ready from %s: %s" % (uid, result)

    def error_callback(self, uid, error):
        if uid != self.uid:
            return
        print "Data error from %s: %s" % (uid, error)

    def fetch(self):
        if self.request is not None:
            # cancel any pending requests
            self.request.cancelled = True
            self.request = None

        self.uid += 1
        self.request = async(slow_method, ["arg1", "arg2"], self.uid,
                             self.ready_callback,
                             self.error_callback)


def slow_method(arg1, arg2):
    print "Starting slow method"
    time.sleep(1)
    return arg1 + arg2


if __name__ == "__main__":
    import sys
    app = QtGui.QApplication(sys.argv)

    obj = ExampleObject()

    dialog = QtGui.QDialog()
    layout = QtGui.QVBoxLayout(dialog)
    button = QtGui.QPushButton("Generate", dialog)
    progress = QtGui.QProgressBar(dialog)
    progress.setRange(0, 0)
    layout.addWidget(button)
    layout.addWidget(progress)
    button.clicked.connect(obj.fetch)
    dialog.show()

    app.exec_()
    app.deleteLater() # avoids some QThread messages in the shell on exit
    # cancel all running tasks avoid QThread/QTimer error messages
    # on exit
    Request.shutdown()

When exiting the application you'll want to make sure you cancel all of the tasks or the application will hang until every scheduled task has completed

View more solutions

Comments

[gwenger]

Thanks this definitely looks useful. If I do it using either the QThread or QObject method, can I add in more of my own signals the same way you made the self.finished signal? For instance, if instead of just printing the count, I want to display the value of count in a QSpinBox that is part of my gui from the other class.

[aukaost]

Yes, you can add your own signals. One way of doing it could be to emit a signal (e.g. pyqtSignal(int)) with the updated value and connect to it from your GUI class to update the QSpinBox accordingly.

[ilpoldo]

I'm not sure that the usingMoveToThread solution is working in my case, I've tried uncommenting usingMoveToThread() and commenting usingQThread, but when I run the script Increasing is never printed out to the terminal.

[aukaost]

I just tried it again using Python 2.7.4 and PyQt4-4.10 and the script still seems to work for me. I'm not sure what your problem could be.

[Matt]

I'm having the same problem. I'm using PyQt4 4.6.2, Python 2.6 and Qt 4.6.2. However, I have an alternate version of PyQt and Qt installed as well, PyQt 4.7.4 and Qt 4.7.4 and it works there. I'm guessing there is a bug in either PyQt or Qt that wasnt addressed until 4.7 or so. This really bites because I need to use the moveToThread way in order to process events in my thread.

[Matt]

I found a strange way to fix it with PyQt 4.6. It seems that QThread::run() is not being called correctly (my guess is this has something to do with QThread::run() no longer being a pure virtual function). This is going to sound dumb but to fix it, just create your own subclass of QThread, reimplement run() and fill in QThread.run(self). Thats it, and it magically works

[Ciasto piekarz]

@aukaost I am not sure why is it just me getting these error executing the above code TypeError: exit() has no overload that is compatible with finished() while calling the functions usingQThread and usingQRunnable..

[Trilarion]

Very nice. This should be the accepted answer nowadays!

[simotek]

From another Qt Dev woboq.com/blog/qthread-you-were-not-doing-so-wrong.html subclassing QThread is perfictly acceptable when just implementing the run method.

[astrojuanlu]

The QMetaObject is ugly as hell

[dashesy]

appears to be a bug in PyQt 4.6 can you point to the issue, so we know if/when it is fixed.

[Michael Scheper]

Upvoting this for the link to the Posch article, since it clears up the conflicting views.

[Matt]

Yeah, the QMetaObject is ugly as sin, but under the hood that's what Qt's signals and slots utilize. This allows the caller to effectively add a message to the Worker's queue of signals in a thread safe manner. Check out my answer for using QRunnables as well. It may have its own blemishes too but it's very powerful in getting asynchronous behavior in your PyQt app. It also doesn't use QMetaObject

[three_pineapples]

For those who are unaware, see this question for why the use of the pyqtSlot()` decorator and specific order of signal connection outlined in this answer matters.

[Neil G]

Inheriting from qthread is fine apparently: woboq.com/blog/qthread-you-were-not-doing-so-wrong.html

[Neil G]

Inheriting from qthread is fine apparently: woboq.com/blog/qthread-you-were-not-doing-so-wrong.html

[Matt]

It's fine to inherit from QThread, but there are drawbacks. One of which is you don't get an event loop (without some work at least). So things like networking that require an event loop don't work well or at all. In general there are a lot of ways to pull off concurrency in Qt/PyQt, it really just depends on what problem you're trying to solve. My favorite way is currently in a separate answer I added closer to the bottom of this page on QRunnable and QThreadPool. Using that you can pass in just about any method and get asynchronous behavior out of it

[Phil]

This is an excellent answer--THANK YOU! There is a blog post Multithreading PyQt applications with QThreadPool that has a similar approach.

[Cecil Curry]

Do not subclass QThread. Doing so is always the wrong thing to do, because the QThread object lives in the parent thread; ergo, slots signalled on this object run in the parent thread rather than the actual thread encapsulated by the QThread object. Signal-slot connections are the entire basis for thread synchronization in Qt. Ergo, subclassing QThread obstructs synchronization. See also: How To Really, Truly Use QThreads; The Full Explanation, the seminal article on the subject.

[Cecil Curry]

No, this should not be the accepted answer. There's absolutely no valid reason to leverage QMetaObject-based hacks prefixed by an entire comment wall of hand-waving when you could simply define proper signal-slot connections instead. </sigh>

[Cecil Curry]

Matthew is entirely correct. Unless you absolutely think you know what you're doing (e.g., you've memorized the QSemaphore API by heart), do not subclass QThread. Either (A) define QObject-based worker classes providing public control slots (e.g., start(), stop(), pause()) and move those workers into a standard QThread object or (B) use QConcurrent.

[cxs1031]

Which one do you suggest for general usage?

[Cerno]

Your link is outdated, you may want to update it: blog.qt.io/blog/2010/06/17/youre-doing-it-wrong

[Skybbles]

You are a godsend, thank you, thank you, thank you for emphasizing the fact that the Worker and Thread needed to have no parent! I was dealing with QThread: Destroyed while thread is still running for the past three hours, then I read this and it clicked!

[Sator]

dude...literal god

[fooforever]

Why don't you pass a parent to the worker object and the thread here? That's what I was doing wrong because that's the C++ way! I can't find any explanation for why you don't do this in python.

[Flying Thunder]

I am a bit confused which of all those options is the right way for me. I have two radio buttons, and a big .connect() on one of them. What do i need to use to get the button to visually change after being pressed, without having to wait for the function to finish?

[Winand]

@CecilCurry woboq.com/blog/qthread-you-were-not-doing-so-wrong.html If I get this right, it's ok to subclass QThread if you do not use slots within your new class

[Flying Thunder]

how do i use this solution when i want the onIntReady function to be in another module, and want to pass input to it?