How to catch exceptions in workers in Multiprocessing

Question

I'm working with the multiprocessing module in Python (2.7.3) and want to debug some stuff going on in my workers. However, I seem to not be able to catch any exceptions in the worker threads.

A minimal example:

import multiprocessing as mp


a=[1]
def worker():
    print a[2]


def pool():
    pool = mp.Pool(processes=1)
    pool.apply_async(worker, args = ())
    pool.close()
    pool.join()
    print "Multiprocessing done!"


if __name__ == '__main__':
    pool()

This is expected to raise an IndexError, but my output only is

    Multiprocessing done!

Is there a way to show me all exceptions occuring in the worker threads without manually raising my own?

Solution 1

The error is not raised unless you call get method of AsyncResult (the return value of the apply_async):

According to the AsyncResult.get documentation:

Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().

def pool():
    pool = mp.Pool(processes=1)
    result = pool.apply_async(worker, args=())
    result.get() # <------------
    pool.close()
    pool.join()
    print "Multiprocessing done!"

Solution 2

I think falsetru gave you what you need. I'd just like to expand a little more.

If it's important for you to get not only the error but the original context (i.e. to know that the exception occurred on line 1 of worker()) then you can check this nice post by Ned Batchelder which explains how to reraise exceptions with their original context.

That doesn't work for mp.Pool so it's just in case you need something more. This SO Question covers your question using more explicit multiprocessing techniques instead of mp.Pool.

Comments

[Dschoni]

Wow, THAT was fast. Thanks.

[Dschoni]

Sorry, I guess that was me by accident. I come back to this answer so often...However my downvote can't be canceled for 2 hours. Sorry.

[Masoud Rahimi]

If I don't want to wait for the process to finish and still wants to get exceptions, what can I do? I mean using result.get() would block the code until the process is finished how to escape that?

[falsetru]

AFAIK, exception -> cause the process termination. You need to wait the process end.