how do I use key word arguments with python multiprocessing pool apply_async

Question

I'm trying to get to grips with pythons multiprocessing module, specifically the apply_async method of Pool. I'm trying to call a function with arguments and keyword arguments. If I call the function without kwargs it's fine but when I try to add in a keyword argument I get: TypeError: apply_async() got an unexpected keyword argument 'arg2' Below is the test code that I'm running

#!/usr/bin/env python
import multiprocessing
from time import sleep
def test(arg1, arg2=1, arg3=2):
    sleep(5)

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    for t in range(1000):
        pool.apply_async(test, t, arg2=5)
    pool.close()
    pool.join()

How can I call the function so that it accepts keyword arguments?

Solution 1

Pass the keyword args in a dictionary (and the positional arguments in a tuple):

pool.apply_async(test, (t,), dict(arg2=5))

Solution 2

original answer: python multiprocessing with boolean and multiple arguments

apply_async has args and kwds keyword arguments which you could use like this:

res = p.apply_async(testFunc, args=(2, 4), kwds={'calcY': False})

Solution 3

Janne's answer didn't work for me in python 2.7.11 (not sure why). The function test() was receiving the key (arg2), not the value (5).

I fixed this by creating a wrapper around test:

def test2(argsDict):
    test(**argsDict)

Then calling

pool.apply_async(test2, (t,), [dict(arg2=5)])

Comments

[Ravindranath Akila]

The thing is, if it is a wrapper, we don't need keyword arguments :) I too confirm the issue of test receiving the key