multiprocessing: map vs map_async

76,574

There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_async only differ with respect to blocking. map_async is non-blocking where as map is blocking

So let's say you had a function

from multiprocessing import Pool
import time

def f(x):
    print x*x

if __name__ == '__main__':
    pool = Pool(processes=4)
    pool.map(f, range(10))
    r = pool.map_async(f, range(10))
    # DO STUFF
    print 'HERE'
    print 'MORE'
    r.wait()
    print 'DONE'

Example output:

pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row. r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.

76,574

Author by

aman

I am a Machine learning enthusiast.

Updated on January 17, 2020

Comments

aman over 4 years
What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes?

So is it wrong to presume both are running asynchronous and parallel?
```
def f(x):
   return 2*x

p=Pool(4)
l=[1,2,3,4]
out1=p.map(f,l)
#vs
out2=p.map_async(f,l)
```
aman about 8 years

ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
quikst3r about 8 years

Not quite. You'll notice map will execute in order, but map_async doesn't
HBeel over 7 years

Should there be a print 'DONE' after r.wait()?
quikst3r over 7 years

Yes there should be!
webelo over 6 years

If above example doesn't return different results for map and map_async on first run, try setting range(500) or something large.
Jirka over 5 years

stackoverflow.com/questions/35708371/…
ScipioAfricanus about 5 years

Hello, will this work with this solution: stackoverflow.com/questions/5442910/… ? I am attempting to pass multiple args to the function.
Catbuilts almost 5 years

@quikst3r: For me they both execute in random order. I change func f() to print(x), and the parameters: "pool.map(f, range(11,20)); r = pool.map_async(f, range(10))"; After run the code, it return: "11 12 13 15 16 17 18 14 19 HERE 0 MORE 2 1 3 4 5 6 7 8 9 DONE"
v.tralala over 4 years

If you don't see HERE and MORE and random positions you don't need to set range to a very high number and then keep scrolling for the prints. For me it was enough to set a time.sleep(0.0002) right before the print part And increase range a little bit to 100
dano about 3 years

@Catbuilts Yeah, there is no difference in ordering between map_async and map, @quikst3r was mistaken when he said that. map is actually implemented internally as map_async(...).get().