ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool
When using map
from concurrent.futures
, each element from the iterable is submitted separately to the executor, which creates a Future
object for each call. It then returns an iterator which yields the results returned by the futures.
Future
objects are rather heavyweight, they do a lot of work to allow all the features they provide (like callbacks, ability to cancel, check status, ...).
Compared to that, multiprocessing.Pool
has much less overhead. It submits jobs in batches (reducing IPC overhead), and directly uses the result returned by the function. For big batches of jobs, multiprocessing is definitely the better options.
Futures are great if you want to sumbit long running jobs where the overhead isn't that important, where you want to be notified by callback or check from time to time to see if they're done or be able to cancel the execution individually.
Personal note:
I can't really think of much reasons to use Executor.map
- it doesn't give you any of the features of futures - except for the ability to specify a timeout. If you're just interested in the results, you're better off using one of multiprocessing.Pool
's map functions.
astrojuanlu
I'm an Aerospace Engineer, with interest in scientific computing, applied math, education, and outreach. I work at Read the Docs on documentation, espcially for science project https://readthedocs.org/ I'm the main developer of poliastro, a Python library for Astrodynamics http://docs.poliastro.space/
Updated on February 01, 2020Comments
-
astrojuanlu about 4 years
I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I've noticed that, almost with identical code, using the Pool from concurrent.futures is way slower than using multiprocessing.Pool.
This is the version using multiprocessing:
def hard_work(n): # Real hard work here pass if __name__ == '__main__': from multiprocessing import Pool, cpu_count try: workers = cpu_count() except NotImplementedError: workers = 1 pool = Pool(processes=workers) result = pool.map(hard_work, range(100, 1000000))
And this is using concurrent.futures:
def hard_work(n): # Real hard work here pass if __name__ == '__main__': from concurrent.futures import ProcessPoolExecutor, wait from multiprocessing import cpu_count try: workers = cpu_count() except NotImplementedError: workers = 1 pool = ProcessPoolExecutor(max_workers=workers) result = pool.map(hard_work, range(100, 1000000))
Using a naïve factorization function taken from this Eli Bendersky article, these are the results on my computer (i7, 64-bit, Arch Linux):
[juanlu@nebulae]─[~/Development/Python/test] └[10:31:10] $ time python pool_multiprocessing.py real 0m10.330s user 1m13.430s sys 0m0.260s [juanlu@nebulae]─[~/Development/Python/test] └[10:31:29] $ time python pool_futures.py real 4m3.939s user 6m33.297s sys 0m54.853s
I cannot profile these with the Python profiler because I get pickle errors. Any ideas?
-
astrojuanlu over 10 yearsThank you very much for your answer! Probably the submitting in batches is the key thing here.
-
dano about 9 yearsFor what it's worth, in Python 3.5,
ProcessPoolExecutor.map
will accept achunksize
keyword argument, which will alleviate the IPC overhead issue somewhat. See this bug for more info. -
Kieleth over 7 yearsAlso, In Python 3.2 you can set maxtasksperchild for a multiprocess Pool which, in my case, helped to clean up resources after each worker finished its workload. link
-
Ciprian Tomoiagă over 5 years
-
astrojuanlu about 4 yearsIt looks like the bug @Ciprian mentions is still open and there are some unfinished attempts to fix it, the latest one being github.com/python/cpython/pull/16103