How can I use pickle to save a dict?
Solution 1
Try this:
import pickle
a = {'hello': 'world'}
with open('filename.pickle', 'wb') as handle:
pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)
with open('filename.pickle', 'rb') as handle:
b = pickle.load(handle)
print a == b
Solution 2
import pickle
your_data = {'foo': 'bar'}
# Store data (serialize)
with open('filename.pickle', 'wb') as handle:
pickle.dump(your_data, handle, protocol=pickle.HIGHEST_PROTOCOL)
# Load data (deserialize)
with open('filename.pickle', 'rb') as handle:
unserialized_data = pickle.load(handle)
print(your_data == unserialized_data)
The advantage of HIGHEST_PROTOCOL
is that files get smaller. This makes unpickling sometimes much faster.
Important notice: The maximum file size of pickle is about 2GB.
Alternative way
import mpu
your_data = {'foo': 'bar'}
mpu.io.write('filename.pickle', data)
unserialized_data = mpu.io.read('filename.pickle')
Alternative Formats
- CSV: Super simple format (read & write)
- JSON: Nice for writing human-readable data; VERY commonly used (read & write)
- YAML: YAML is a superset of JSON, but easier to read (read & write, comparison of JSON and YAML)
- pickle: A Python serialization format (read & write)
- MessagePack (Python package): More compact representation (read & write)
- HDF5 (Python package): Nice for matrices (read & write)
- XML: exists too *sigh* (read & write)
For your application, the following might be important:
- Support by other programming languages
- Reading / writing performance
- Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
Solution 3
# Save a dictionary into a pickle file.
import pickle
favorite_color = {"lion": "yellow", "kitty": "red"} # create a dictionary
pickle.dump(favorite_color, open("save.p", "wb")) # save it into a file named save.p
# -------------------------------------------------------------
# Load the dictionary back from the pickle file.
import pickle
favorite_color = pickle.load(open("save.p", "rb"))
# favorite_color is now {"lion": "yellow", "kitty": "red"}
Solution 4
In general, pickling a dict
will fail unless you have only simple objects in it, like strings and integers.
Python 2.7.9 (default, Dec 11 2014, 01:21:43)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type(globals())
<type 'dict'>
>>> import pickle
>>> pik = pickle.dumps(globals())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>>
Even a really simple dict
will often fail. It just depends on the contents.
>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function <lambda> at 0x102178668>: it's not found as __main__.<lambda>
However, if you use a better serializer like dill
or cloudpickle
, then most dictionaries can be pickled:
>>> import dill
>>> pik = dill.dumps(d)
Or if you want to save your dict
to a file...
>>> with open('save.pik', 'w') as f:
... dill.dump(globals(), f)
...
The latter example is identical to any of the other good answers posted here (which aside from neglecting the picklability of the contents of the dict
are good).
Solution 5
Simple way to dump a Python data (e.g. dictionary) to a pickle file.
import pickle
your_dictionary = {}
pickle.dump(your_dictionary, open('pickle_file_name.p', 'wb'))
Chachmu
Updated on July 08, 2022Comments
-
Chachmu almost 2 years
I have looked through the information that the Python docs give, but I'm still a little confused. Could somebody post sample code that would write a new file then use pickle to dump a dictionary into it?