How does the callback function work in multiprocessing map_async?
45,978
Callback is called once with the result ([[0], [0, 1]]
) if you use map_async.
>>> from multiprocessing import Pool
>>> def myfunc(x):
... return [i for i in range(x)]
...
>>> A = []
>>> def mycallback(x):
... print('mycallback is called with {}'.format(x))
... A.extend(x)
...
>>> pool=Pool()
>>> r = pool.map_async(myfunc, (1,2), callback=mycallback)
>>> r.wait()
mycallback is called with [[0], [0, 1]]
>>> print(A)
[[0], [0, 1]]
Use apply_async
if you want callback to be called for each time.
pool=Pool()
results = []
for x in (1,2):
r = pool.apply_async(myfunc, (x,), callback=mycallback)
results.append(r)
for r in results:
r.wait()
Author by
user2727768
Updated on July 09, 2022Comments
-
user2727768 almost 2 years
It cost me a whole night to debug my code, and I finally found this tricky problem. Please take a look at the code below.
from multiprocessing import Pool def myfunc(x): return [i for i in range(x)] pool=Pool() A=[] r = pool.map_async(myfunc, (1,2), callback=A.extend) r.wait()
I thought I would get
A=[0,0,1]
, but the output isA=[[0],[0,1]]
. This does not make sense to me because if I haveA=[]
,A.extend([0])
andA.extend([0,1])
will give meA=[0,0,1]
. Probably the callback works in a different way. So my question is how to getA=[0,0,1]
instead of[[0],[0,1]]
?