How to get the result of multiprocessing.Pool.apply_async

Question

I want to get the result of the function run by Pool.apply_async in Python.

How to assign the result to a variable in the parent process? I tried to use callback but it seems complicated.

Solution 1

The solution is very simple:

import multiprocessing

def func():
    return 2**3**4

p = multiprocessing.Pool()
result = p.apply_async(func).get()
print(result)

Since Pool.apply_async() returns an AsyncResult, you can simply get the result from the AsyncResult.get() method.

Hope this helps!

Solution 2

Well an easy way would be to have a helper class like this:

class Result():
    def __init__(self):
        self.val = None

    def update_result(self, val):
        self.val = val

result = Result()

def f(x):
    return x*x

pool.apply_async(f, (10,), callback=result.update_result)

When the thread runs and calculates the result it will call your callback which will update the result.val.

In any case you need to check that the thread has finished.

Comments

[THN]

Thanks. Should result be global variable? I mean is it accessed in the subprocess?

[Giannis Spiliopoulos]

Not necessarily. However, result should be in the same scope as your call to pool.apply_async.

[THN]

Thanks. This is clean. Can get() return arbitrary object type?

[tdelaney]

@thn - mostly... the subprocess can only return picklable objects. The pool needs to serialize the data to pass it back to the parent and that's how it does it.

[THN]

I see. +1 for mentioning serializing, so I need to think about the overhead of serializing.

[THN]

I think this answer is the pythonic way to do it, so accepted answer. Thank you all.

[THN]

Note that AsyncResult.get() will return the result when it arrives, so it will block the parent process. To parallelize with multiple apply_async calls, better use async_result = p.apply_async(func) then result = async_result.get() after p.close(); p.join().

[linusg]

Thanks for the information, you may adapt your code as you want :) This is just for the general way of doing it.

[Jamie Marshall]

@THN - won't that prevent processing the result until all results are ready?

[Jamie Marshall]

@GiannisSpiliopoulos - what does "not necessarily" mean? The only two scenarios I can image here, is 1. function is called in the sub process, 2. pool.apply_async turns the synchronous caller into an asynchronous awaitable.