Python: How to make program wait till function's or method's completion

Question

Often there is a need for the program to wait for a function to complete its work. Sometimes it is opposite: there is no need for a main program to wait. I've put a simple example. There are four buttons. Clicking each will call the same calculate() function. The only difference is the way the function is called.

1. "Call Directly" button calls calculate() function directly. Since there is a 'Function End' print out it is evident that the program is waiting for the calculate function to complete its job.
2. "Call via Threading" calls the same function this time using threading mechanism. Since the program prints out ': Function End' message immidiately after the button is presses I can conclude the program doesn't wait for calculate() function to complete. How to override this behavior? How to make program wait till calculate() function is finished?

3. "Call via Multiprocessing" buttons utilizes multiprocessing to call calculate() function. Just like with threading multiprocessing doesn't wait for function completion. What statement we have to put in order to make it wait?

4. "Call via Subprocess" buttons doesn't do anything since I didn't figure out the way to hook subprocess to run internal script function or method. It would be interesting to see how to do it...

Example:

from PyQt4 import QtCore, QtGui    
app = QtGui.QApplication(sys.argv)


def calculate(listArg=None):
    print '\n\t Starting calculation...'
    m=0
    for i in range(50000000):
        m+=i
    print '\t ...calculation completed\n'


class Dialog_01(QtGui.QMainWindow):
    def __init__(self):
        super(Dialog_01, self).__init__()

        myQWidget = QtGui.QWidget()
        myBoxLayout = QtGui.QVBoxLayout()       

        directCall_button = QtGui.QPushButton("Call Directly")
        directCall_button.clicked.connect(self.callDirectly)      
        myBoxLayout.addWidget(directCall_button) 

        Button_01 = QtGui.QPushButton("Call via Threading")
        Button_01.clicked.connect(self.callUsingThreads)
        myBoxLayout.addWidget(Button_01)        

        Button_02 = QtGui.QPushButton("Call via Multiprocessing")
        Button_02.clicked.connect(self.callUsingMultiprocessing)
        myBoxLayout.addWidget(Button_02) 

        Button_03 = QtGui.QPushButton("Call via Subprocess")
        Button_03.clicked.connect(self.callUsingSubprocess)
        myBoxLayout.addWidget(Button_03) 


        myQWidget.setLayout(myBoxLayout)
        self.setCentralWidget(myQWidget)
        self.setWindowTitle('Dialog 01')

    def callUsingThreads(self):
        print '------------------------------- callUsingThreads() ----------------------------------'
        import threading
        self.myEvent=threading.Event()
        self.c_thread=threading.Thread(target=calculate)
        self.c_thread.start()  

        print "\n\t\t : Function End"


    def callUsingMultiprocessing(self):
        print '------------------------------- callUsingMultiprocessing() ----------------------------------'
        from multiprocessing import Pool

        pool = Pool(processes=3)
        try: pool.map_async( calculate, ['some'])
        except Exception, e: print e 

        print "\n\t\t : Function End"


    def callDirectly(self):
        print '------------------------------- callDirectly() ----------------------------------'
        calculate()
        print "\n\t\t : Function End"


    def callUsingSubprocess(self):
        print '------------------------------- callUsingSubprocess() ----------------------------------'
        import subprocess 
        print '-missing code solution'
        print "\n\t\t : Function End"


if __name__ == '__main__':
    dialog_1 = Dialog_01()
    dialog_1.show()
    dialog_1.resize(480,320)
    sys.exit(app.exec_())

Solution 1

Use a queue: each thread when completed puts the result on the queue and then you just need to read the appropriate number of results and ignore the remainder:

#!python3.3
import queue    # For Python 2.x use 'import Queue as queue'
import threading, time, random

def func(id, result_queue):
    print("Thread", id)
    time.sleep(random.random() * 5)
    result_queue.put((id, 'done'))

def main():
    q = queue.Queue()
    threads = [ threading.Thread(target=func, args=(i, q)) for i in range(5) ]
    for th in threads:
        th.daemon = True
        th.start()

    result1 = q.get()
    result2 = q.get()

    print("Second result: {}".format(result2))

if __name__=='__main__':
    main()

Documentation for Queue.get() (with no arguments it is equivalent to Queue.get(True, None):

    Queue.get([block[, timeout]])

Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

How to wait until only the first thread is finished in Python

You can to use .join() method too. what is the use of join() in python threading

Solution 2

I find that using the "pool" submodule within "multiprocessing" works amazingly for executing multiple processes at once within a Python Script.

See Section: Using a pool of workers

Look carefully at "# launching multiple evaluations asynchronously may use more processes" in the example. Once you understand what those lines are doing, the following example I constructed will make a lot of sense.

import numpy as np
from multiprocessing import Pool

def desired_function(option, processes, data, etc...):
    # your code will go here. option allows you to make choices within your script
    # to execute desired sections of code for each pool or subprocess.

    return result_array   # "for example"


result_array = np.zeros("some shape")  # This is normally populated by 1 loop, lets try 4.
processes = 4
pool = Pool(processes=processes)
args = (processes, data, etc...)    # Arguments to be passed into desired function.

multiple_results = []
for i in range(processes):          # Executes each pool w/ option (1-4 in this case).
    multiple_results.append(pool.apply_async(param_process, (i+1,)+args)) # Syncs each.

results = np.array(res.get() for res in multiple_results)  # Retrieves results after
                                                           # every pool is finished!

for i in range(processes):
    result_array = result_array + results[i]  # Combines all datasets!

The code will basically run the desired function for a set number of processes. You will have to carefully make you're function can distinguish between each process (hence why I added the variable "option".) Additionally, it doesn't have to be an array that is being populated in the end, but for my example thats how I used it. Hope this simplifies or helps you better understand the power of multiprocessing in Python!

Comments

[aruisdante]

I think you want to look into the concept of thread's join() method.

[Blckknght]

I'm not sure I understand why you want to use threads or processes if you want the calculation to run synchronously. Why not just run it in the main process/thread?

[alphanumeric]

I just want to make sure I know what the program does when I use or don't use threading, subprocess and multiprocessing. Aside from knowing what and how exactly those modules methods do it could be helpful to know how to control them... for instance, a .join() method (threading module) makes a program to wait till function completes its calculation. I think it could be pretty useful to know..

[alphanumeric]

Thanks!! Still waiting to see how subprocess module could be used with it too!

[Allan Karlson]

Wow that threat saved my day. In python2.7 the queue module does not exist but instead it called Queue :D