Can't pickle Function

Question

So I'm trying to speed up my computation time by doing a little bit multiprocessing

I'm trying to use the pool workers.

At the top of my code I have

import Singal as s
import multiprocessing as mp
def wrapper(Channel):
    Noise_Frequincies = []
    for i in range(1,125):
        Noise_Frequincies.append(60.0*float(i))
    Noise_Frequincies.append(180.0)
    filter1 = s.Noise_Reduction(Sample_Rate,Noise_Frequincies,Channel)
    return filter1

Then when the time comes I use

Both_Channels = [Chan1, Chan2]
results = mp.Pool(2).map(wrapper,Both_Channels)
filter1 = results[0]
filter2 = results[1]

I get the following error

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Chan1 and Chan2 are arrays of my signal, and I'm trying to filter out some noise out of each. I'm new to multiprocessing so I apologize if this is a dumb error

Solution

I marked this question as dup of that Q/A, and to sum up: you can't pickle functions, and that's what you're trying to do when you pass wrapper() to Pool().map(). To pickle functions, you need to use copy_reg, as shown by this example.

" Classes, functions, and methods cannot be pickled -- if you pickle an object, the object's class is not pickled, just a string that identifies what class it belongs to. " (cf doc)

• http://docs.python.org/2/library/copy_reg.html?highlight=copyreg
• http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods

I don't use custom classes, and that's where his issue is (at least as far as I can tell)

no, his problem that he was trying to pickle an instance method, which is relatively close to a function, as both can't be pickled. And the workaround for that A should work for you too.

Though I did not test it…

HTH

Comments

[roippi]

you can pickle top-level functions; all multiprocessing has to do is pickle a string representing the name. I'm pretty sure the pickling error comes from trying to pickle Both_Channels, not wrapper. A bit hard to tell since the traceback is unhelpful.

[roippi]

or actually the pickling error probably comes from trying to ship s.Noise_Reduction objects back through the pipe. Bet that's it.

[zmo]

my feeling is that it's indeed wrapper that is failing (given the exception raised), though you may be right about s.Noise_Reduction being the thing within the function that makes it unpickable.

[Cate Daniel]

Is there an easy way to pickle several functions? Noise reductions also calls several other methods to reduce the noise in my signal. Am I just simply barking up the wrong tree to make things more efficient?

[zmo]

Well, I think, that whatever the real bug is, your problem is that you're trying to move around algorithms between processes instead of moving around only the datasets you want the algorithms to work on. Because the algorithm do not change for the life of the program, you can instanciate easily your processes with them, and get only the data to be pickled between them.