Python Multiprocessing Lib Error (AttributeError: __exit__)

Question

Am getting this error when using the pool.map(funct, iterable):

AttributeError: __exit__

No Explanation, only stack trace to the pool.py file within the module.

using in this way:

with Pool(processes=2) as pool:
   pool.map(myFunction, mylist)
   pool.map(myfunction2, mylist2)

I suspect there could be a problem with the picklability (python needs to pickle, or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.

EDIT: new format of code that produces this error :

def governingFunct(list):
    #some tasks
    def myFunction():
         # function contents
    with closing(Pool(processes=2)) as pool:
         pool.map(myFunction, sublist)
         pool.map(myFunction2, sublist2)

ERROR PRODUCED:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Solution 1

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

For earlier Python versions, you could use contextlib.closing(), but take into account this'll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)

Solution 2

with statement is for object that have __enter__ and __exit__ functions, i.e. Context Manager Types
multiprocessing.Pool is not Context Manager Type. try do the following:

pool = Pool(processes=2)
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)

Comments

[Martijn Pieters]

The question is why the OP thought it was. The answer is that the Python 3 documentation tells you it is. But it also qualifies that as Python 3.3 and up.

[sidewaiise]

Well Sir thank you for such a detailed explanation. You're solution has moved the function a step closer to working - now I am getting this error: PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

[Martijn Pieters]

@sidewaiise: are you trying to use a method on a class perhaps? See Multiprocessing: using Pool.map on a function defined in a class

[sidewaiise]

Heh I was actually just reading that as you posted this. Thanks - I'll have a read and comment shortly.

[sidewaiise]

Not sure I understand the solution on that link - I'm not using classes. It's as simple as I wrote above, except that it is inside another function.... eg: **** Refer to Edited Question ^ **** So not sure.. any ideas?

[Martijn Pieters]

@sidewaiise: functions nested in another function are not picklable. Normally, pickle will store just a name (modulename.functionname) then import the same function on the other side. You cannot do that when you need to call another function to get the function.

[sidewaiise]

Getting there... new error: IndexError: string index out of range. Again no explanation on where the error actually occurs.

[sidewaiise]

(I've taken the Pool method out of the original function also)

[Martijn Pieters]

If you have new problems, better ask a new question. Your original issue here has been solved after all, and we now have a nice question + answer that'll perhaps others in future. No need to muddle this with new, different problems.

[sidewaiise]

True. Have done just now actually, thanks. stackoverflow.com/questions/25974256/…

[paulochf]

Excuse me, but when I'm creating my own "terminating" do I have to do from contextlib import contextmanager ?

[Martijn Pieters]

@paulochf: for some reason the decorator line was missing. Only import it when you actually plan on using it! :-) (and sorry).