Can Python pickle lambda functions?

Question

I have read in a number of threads that Python pickle/cPickle cannot pickle lambda functions. However the following code works, using Python 2.7.6:

import cPickle as pickle

if __name__ == "__main__":
    s = pickle.dumps(lambda x, y: x+y)
    f = pickle.loads(s)
    assert f(3,4) == 7

So what is going on? Or, rather, what is the limit of pickling lambdas?

[EDIT] I think i know why this code runs. I forgot (sorry!) i am running stackless python, which has a form of micro-threads called tasklets executing a function. These tasklets can be halted, pickled, unpickled and continued, so i guess (asked on the stackless mailing list) that it also provides a way to pickle function bodies.

Solution 1

Yes, python can pickle lambda functions… but only if you have something that uses copy_reg to register how to pickle lambda functions -- the package dill loads the copy_reg you need into the pickle registry for you, when you import dill.

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import dill  # the code below will fail without this line
>>> 
>>> import pickle
>>> s = pickle.dumps(lambda x, y: x+y)
>>> f = pickle.loads(s)
>>> assert f(3,4) == 7
>>> f
<function <lambda> at 0x10aebdaa0>

get dill here: https://github.com/uqfoundation

Solution 2

Python can pickle lambdas. We will cover Python 2 and 3 separately as implementation of pickle are different in different Python versions.

• Python 3.6

In Python 3, there is no module named cPickle. We have pickle instead which also doesn't support pickling of lambda functions by default. Let's see it's dispatch table:

>> import pickle
>> pickle.Pickler.dispatch_table
<member 'dispatch_table' of '_pickle.Pickler' objects>

Wait. I tried looking up dispatch_table of pickle not _pickle. _pickle is the alternative and faster C implementation of pickle. But we haven't imported it yet! This C implementation is imported automatically, if it is available, at the end of pure Python pickle module.

# Use the faster _pickle if possible
try:
    from _pickle import (
        PickleError,
        PicklingError,
        UnpicklingError,
        Pickler,
        Unpickler,
        dump,
        dumps,
        load,
        loads
    )
except ImportError:
    Pickler, Unpickler = _Pickler, _Unpickler
    dump, dumps, load, loads = _dump, _dumps, _load, _loads

We are still left with the question of pickling lambdas in Python 3. The answer is you CAN'T with the native pickle or _pickle. You will need to import dill or cloudpickle and use that instead of the native pickle module.

>> import dill
>> dill.loads(dill.dumps(lambda x:x))
<function __main__.<lambda>>

• Python 2.7

pickle uses pickle registry which is nothing but a mapping from type to the function to use for serializing (pickling) objects of that type. You can see pickle registry as:

>> pickle.Pickler.dispatch

{bool: <function pickle.save_bool>,
 instance: <function pickle.save_inst>,
 classobj: <function pickle.save_global>,
 float: <function pickle.save_float>,
 function: <function pickle.save_global>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function pickle.save_dict>,
 builtin_function_or_method: <function pickle.save_global>,
 NoneType: <function pickle.save_none>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 type: <function pickle.save_global>,
 unicode: <function pickle.save_unicode>}

To pickle custom types, Python provides copy_reg module to register our functions. You can read more about it here. By default, copy_regmodule supports pickling of the following additional types:

>> import copy_reg
>> copy_reg.dispatch_table

{code: <function ipykernel.codeutil.reduce_code>,
 complex: <function copy_reg.pickle_complex>,
 _sre.SRE_Pattern: <function re._pickle>,
 posix.statvfs_result: <function os._pickle_statvfs_result>,
 posix.stat_result: <function os._pickle_stat_result>}

Now, type of lambda functions is types.FunctionType. However, the builtin function for this type function: <function pickle.save_global> is not able to serialize lambda functions. Therefore, all third party libraries like dill, cloudpickle, etc override the inbuilt method to serialize lambda functions with some additional logic. Let's import dill and see what it does.

>> import dill
>> pickle.Pickler.dispatch

{_pyio.BufferedReader: <function dill.dill.save_file>,
 _pyio.TextIOWrapper: <function dill.dill.save_file>,
 _pyio.BufferedWriter: <function dill.dill.save_file>,
 _pyio.BufferedRandom: <function dill.dill.save_file>,
 functools.partial: <function dill.dill.save_functor>,
 operator.attrgetter: <function dill.dill.save_attrgetter>,
 operator.itemgetter: <function dill.dill.save_itemgetter>,
 cStringIO.StringI: <function dill.dill.save_stringi>,
 cStringIO.StringO: <function dill.dill.save_stringo>,
 bool: <function pickle.save_bool>,
 cell: <function dill.dill.save_cell>,
 instancemethod: <function dill.dill.save_instancemethod0>,
 instance: <function pickle.save_inst>,
 classobj: <function dill.dill.save_classobj>,
 code: <function dill.dill.save_code>,
 property: <function dill.dill.save_property>,
 method-wrapper: <function dill.dill.save_instancemethod>,
 dictproxy: <function dill.dill.save_dictproxy>,
 wrapper_descriptor: <function dill.dill.save_wrapper_descriptor>,
 getset_descriptor: <function dill.dill.save_wrapper_descriptor>,
 member_descriptor: <function dill.dill.save_wrapper_descriptor>,
 method_descriptor: <function dill.dill.save_wrapper_descriptor>,
 file: <function dill.dill.save_file>,
 float: <function pickle.save_float>,
 staticmethod: <function dill.dill.save_classmethod>,
 classmethod: <function dill.dill.save_classmethod>,
 function: <function dill.dill.save_function>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function dill.dill.save_module_dict>,
 builtin_function_or_method: <function dill.dill.save_builtin_method>,
 module: <function dill.dill.save_module>,
 NotImplementedType: <function dill.dill.save_singleton>,
 NoneType: <function pickle.save_none>,
 xrange: <function dill.dill.save_singleton>,
 slice: <function dill.dill.save_slice>,
 ellipsis: <function dill.dill.save_singleton>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 super: <function dill.dill.save_functor>,
 type: <function dill.dill.save_type>,
 weakcallableproxy: <function dill.dill.save_weakproxy>,
 weakproxy: <function dill.dill.save_weakproxy>,
 weakref: <function dill.dill.save_weakref>,
 unicode: <function pickle.save_unicode>,
 thread.lock: <function dill.dill.save_lock>}

Now, let's try to pickle lambda function.

>> pickle.loads(pickle.dumps(lambda x:x))
<function __main__.<lambda>>

It WORKS!!

In Python 2 we have two versions of pickle -

import pickle # pure Python version
pickle.__file__ # <install directory>/python-2.7/lib64/python2.7/pickle.py

import cPickle # C extension
cPickle.__file__ # <install directory>/python-2.7/lib64/python2.7/lib-dynload/cPickle.so

Now, let's try to pickle lambda with C implementation cPickle.

>> import cPickle
>> cPickle.loads(cPickle.dumps(lambda x:x))
TypeError: can't pickle function objects

What went wrong? Let's see the dispatch table of cPickle.

>> cPickle.Pickler.dispatch_table
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch_table'

The implementation of pickle and cPickle is different. Importing dill makes only Python version of pickle work. The disadvantage of using pickle instead of cPickle is that it can be as much as 1000 times slower than cPickle.

I hope this clears all the doubts.

Solution 3

No, Python can't pickle lambda functions:

>>> import cPickle as pickle
>>> s = pickle.dumps(lambda x,y: x+y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Not sure what you did that succeeded...

Solution 4

Even though it might be obvious I would like to add an other possible solution. As you probably know lambda functions are just anonymous function declarations. If you don't have many lambdas that are used only once and it wouldn't add much noise to your code you could just name your lambda and pass the name of it (without the parentheses) like this:

import cPickle as pickle

def addition(x, y):
    return x+y


if __name__ == "__main__":
    s = pickle.dumps(addition)
    f = pickle.loads(s)
    assert f(3,4) == 7

The name also adds more semantic and you wouldn't need an additional dependency like Dill. But only do that if that outweighs the added noise of the additional function(s).

Solution 5

what worked for me (windows 10, python 3.7) was to pass a function instead of a lambda function:

def merge(x):
    return Image.merge("RGB", x.split()[::-1])

transforms.Lambda(merge)

instead of:

transforms.Lambda(lambda x: Image.merge("RGB", x.split()[::-1]))

no dill or cPickel needed.

View more solutions

Comments

[Ramast]

I've tried on python3 In [1]: import dill In [2]: import pickle In [3]: pickle.dumps(lambda x: (x+1, x+2)) ------------------------------------------------------------‌​--------------- PicklingError Traceback (most recent call last) <ipython-input-3-924e2f4cc7e0> in <module>() ----> 1 pickle.dumps(lambda x: (x+1, x+2)) PicklingError: Can't pickle <function <lambda> at 0x7f08ee40ca60>: attribute lookup <lambda> on main failed. It only works if you import dill as pickle

[Ramast]

I don't know why this comment is down voted. pickle can't serialize lambdas only dill package can

[Mike McKerns]

@Ramast: you are correct -- in python3, you currently have to import dill as pickle. In python2, what I have above works either way you do it.

[Mike McKerns]

@CharlieParker: Can you elaborate? The above should work for most "arbitrary" functions.

[naught101]

Why can't python pickle lambdas?

[Pavel Vlasov]

The answer is incorrect for Python 3.6 as it is - use dill.dumps() and dill.loads() instead.

[Mike McKerns]

@PavelVlasov: your comment is a duplicate of the initial comment on the answer.

[Ciprian Tomoiagă]

@MikeMcKerns can you please expand on the differences between Py3 and Py2 version of dill ? Is dill not able to hijack the pickle module anymore? I was hoping that importing dill before rq (which internally uses pickle), I can make rq work with lambdas

[Mike McKerns]

@CiprianTomoiagă: The primary difference is that in python 2, there was pickle (python) and cPickle (C) -- and in python 3, it's now pickle (python) and _pickle (C)... however, pickle.dump (and dumps) uses _pickle (i.e. C)... and dill can currently only inject new methods into the python pickle registry. So, just importing dill doesn't work like it does in python 2. Note that there is pickle._dump (and _dumps), which does use the python registry... so that works as in python 2. Unfortunately most packages don't fall back to _dump, when dump fails.

[EliadL]

This answer should be accepted. It nicely explains the usability and limitations with the dill package in each python version. Well done!

[Charlie Parker]

@MikeMcKerns does it work if the code that has arbitrary pytorch (and data science) code? I am trying to pickle a pytorch neural net work pointer to lambda functions.

[Charlie Parker]

like in this case: stackoverflow.com/questions/61510810/…

[Mike McKerns]

it should. I answered your question directly at the above link