How to pass and run a callback method in Python

Question

I have a Manager (main thread), that creates other Threads to handle various operations. I would like my Manager to be notified when a Thread it created ends (when run() method execution is finished).

I know I could do it by checking the status of all my threads with the Thread.isActive() method, but polling sucks, so I wanted to have notifications.

I was thinking of giving a callback method to the Threads, and call this function at the end of the run() method:

class Manager():
    ...
    MyThread(self.on_thread_finished).start() # How do I pass the callback

    def on_thread_finished(self, data):
        pass
    ...

class MyThread(Thread):
    ...
    def run(self):
        ....
        self.callback(data) # How do I call the callback?
    ...

Thanks!

Solution 1

The thread can't call the manager unless it has a reference to the manager. The easiest way for that to happen is for the manager to give it to the thread at instantiation.

class Manager(object):
    def new_thread(self):
        return MyThread(parent=self)
    def on_thread_finished(self, thread, data):
        print thread, data

class MyThread(Thread):

    def __init__(self, parent=None):
        self.parent = parent
        super(MyThread, self).__init__()

    def run(self):
        # ...
        self.parent and self.parent.on_thread_finished(self, 42)

mgr    = Manager()
thread = mgr.new_thread()
thread.start()

If you want to be able to assign an arbitrary function or method as a callback, rather than storing a reference to the manager object, this becomes a bit problematic because of method wrappers and such. It's hard to design the callback so it gets a reference to both the manager and the thread, which is what you will want. I worked on that for a while and did not come up with anything I'd consider useful or elegant.

Solution 2

Anything wrong with doing it this way?

from threading import Thread

class Manager():
    def Test(self):
        MyThread(self.on_thread_finished).start()

    def on_thread_finished(self, data):
        print "on_thread_finished:", data

class MyThread(Thread):
    def __init__(self, callback):
        Thread.__init__(self)
        self.callback = callback

    def run(self):
        data = "hello"
        self.callback(data)

m = Manager()
m.Test() # prints "on_thread_finished: hello"

Solution 3

If you want the main thread to wait for children threads to finish execution, you are probably better off using some kind of synchronization mechanism. If simply being notified when one or more threads has finished executing, a Condition is enough:

import threading

class MyThread(threading.Thread):
    def __init__(self, condition):
        threading.Thread.__init__(self)
        self.condition = condition

    def run(self):
        print "%s done" % threading.current_thread()
        with self.condition:
            self.condition.notify()


condition = threading.Condition()
condition.acquire()

thread = MyThread(condition)
thread.start()

condition.wait()

However, using a Queue is probably better, as it makes handling multiple worker threads a bit easier.

Comments

[ThorSummoner]

I am interested in a non-threaded callback method example.

[Bharathwaaj]

You cannot pass the local method on_thread_finished to the thread as callback because the local method takes two arguments. When the callback is called from the thread, it will give only one argument (data)

[Shariq]

I know this is late, but for anyone else who is about to spend an hour rewriting a large chunk of code: Bharathwaaj's comment is completely incorrect and this answer works perfectly.

[Be Brave Be Like Ukraine]

@Shariq, the problem is that on_thread_finished still runs in the child thread, not in the Manager. You can't make Manager do it because Manager is doing something else at this time. To see the problem, just make Test() hang in a dead loop after calling MyThread(). Alternatively, define a local variable and try using it within on_thread_finished(), e.g. number_of_threads_running -=1

[Jason]

This won't work if the manager is in a tkinter GUI, you can't call the GUI's functions from a work thread

[iliis]

Shouldn't the last line be thread.start()? Otherwise, no actual multithreading will occur and MyThread.run() gets executed like any other function. But if you actually run MyThread.start() and create a new thread, self.parent.on_thread_finished(self, 42) will still be executed in the new thread's context instead of the main thread. You will need some kind of synchronization, like a Queue, so @jonathan-lillesæter is actually correct.

[Josh Noe]

New to Python. Isn't this a misuse of inheritance? MyThread doesn't seem like a logical child of Manager.

[Sindarus]

I think it's worth noting that using this solution will execute the callback function inside the secondary thread, not the main thread. Also, what is the use of evaluating self.parent in function run ? Is this to prevent self.parent.on_thread_finished from being called if parent is None ? Why would you do that instead of using an if statement ? This is confusing...