Storing Python dictionaries

Question

I'm used to bringing data in and out of Python using CSV files, but there are obvious challenges to this. Are there simple ways to store a dictionary (or sets of dictionaries) in a JSON or pickle file?

For example:

data = {}
data ['key1'] = "keyinfo"
data ['key2'] = "keyinfo2"

I would like to know both how to save this, and then how to load it back in.

Solution 1

Pickle save:

try:
    import cPickle as pickle
except ImportError:  # Python 3.x
    import pickle

with open('data.p', 'wb') as fp:
    pickle.dump(data, fp, protocol=pickle.HIGHEST_PROTOCOL)

See the pickle module documentation for additional information regarding the protocol argument.

Pickle load:

with open('data.p', 'rb') as fp:
    data = pickle.load(fp)

---

JSON save:

import json

with open('data.json', 'w') as fp:
    json.dump(data, fp)

Supply extra arguments, like sort_keys or indent, to get a pretty result. The argument sort_keys will sort the keys alphabetically and indent will indent your data structure with indent=N spaces.

json.dump(data, fp, sort_keys=True, indent=4)

JSON load:

with open('data.json', 'r') as fp:
    data = json.load(fp)

Solution 2

Minimal example, writing directly to a file:

import json
json.dump(data, open(filename, 'wb'))
data = json.load(open(filename))

or safely opening / closing:

import json
with open(filename, 'wb') as outfile:
    json.dump(data, outfile)
with open(filename) as infile:
    data = json.load(infile)

If you want to save it in a string instead of a file:

import json
json_str = json.dumps(data)
data = json.loads(json_str)

Solution 3

Also see the speeded-up package ujson:

import ujson

with open('data.json', 'wb') as fp:
    ujson.dump(data, fp)

Solution 4

To write to a file:

import json
myfile.write(json.dumps(mydict))

To read from a file:

import json
mydict = json.loads(myfile.read())

myfile is the file object for the file that you stored the dict in.

Solution 5

If you want an alternative to pickle or json, you can use klepto.

>>> init = {'y': 2, 'x': 1, 'z': 3}
>>> import klepto
>>> cache = klepto.archives.file_archive('memo', init, serialized=False)
>>> cache        
{'y': 2, 'x': 1, 'z': 3}
>>>
>>> # dump dictionary to the file 'memo.py'
>>> cache.dump() 
>>> 
>>> # import from 'memo.py'
>>> from memo import memo
>>> print memo
{'y': 2, 'x': 1, 'z': 3}

With klepto, if you had used serialized=True, the dictionary would have been written to memo.pkl as a pickled dictionary instead of with clear text.

You can get klepto here: https://github.com/uqfoundation/klepto

dill is probably a better choice for pickling then pickle itself, as dill can serialize almost anything in python. klepto also can use dill.

You can get dill here: https://github.com/uqfoundation/dill

The additional mumbo-jumbo on the first few lines are because klepto can be configured to store dictionaries to a file, to a directory context, or to a SQL database. The API is the same for whatever you choose as the backend archive. It gives you an "archivable" dictionary with which you can use load and dump to interact with the archive.

View more solutions

Comments

[Greg Hewgill]

Have you read the documentation for the json or pickle standard modules?

[Martin Thoma]

See Save a dictionary to a file (alternative to pickle) in Python?

[agf]

If you want to store a whole dict, or load a whole dict, json is more convenient. shelve is only better for accessing one key at a time.

[Admin]

you are ware that json has that take files as arguments and write directly to them?

[Jonathanb]

JSON does dictionaries natively (though they obviously don't behave exactly as a python dictionary does while in memory, for persistence purposes, they are identical). In fact, the foundational unit in json is the "Object", which is defined as { <string> : <value>}. Look familiar? The json module in the standard library supports every Python native type and can easily be extended with a minimal knowledge of json to support user-defined classes. The JSON homepage completely defines the language in just over 3 printed pages, so it's easy to absorb/digest quickly.

[Steve Jessop]

It's worth knowing about the third argument to pickle.dump, too. If the file doesn't need to be human-readable then it can speed things up a lot.

[Dror]

Seems like this doesn't work out-of-the-box with 3.x. Which part has to be casted?

[Martijn Pieters]

Why are you using binary mode for the JSON files here? JSON is a text format.

[Martijn Pieters]

@Dror: The json library produces text (type str) data, so the files should not be opened in binary mode.

[Niklas R]

json.dump(myfile) and json.load(myfile)

[Marty]

@MartijnPieters, and Dror: I'm fairly certain that was a poorly executed copy and paste. Fixed in the most recent edit, thanks for pointing out the problem.

[juliusmh]

If you add sort_keys and indent arguments to the dump call you get a much prettier result. e.g.: json.dump(data, fp, sort_keys=True, indent=4). More info can be found here

[Martin Thoma]

You should probably use pickle.dump(data, fp, protocol=pickle.HIGHEST_PROTOCOL)

[Matteo]

@Marty - Is there any advantage of using pickle vs json? json is a readable file instead of a byte stream; size of resulting file, etc?

[dvdhns]

@Matteo Json won't work with dict that have tuples as keys { (1,2): "abc" }, but they are readable by people. Pickle will save dicts that have tuples as keys, but they are not as readble by people as Json.

[Melroy van den Berg]

For python 3, use import pickle

[JohnAndrews]

Is it only me or is saving the json really slow? Trying to store approx 100mb json.

[haneulkim]

There are multiple ways of importing a dictionary. Importing variable from different file, using json, and using pickle. What are the differences?

[Alon Samuel]

I've tried to check which is better in memory usage, for me pickle was 5 kb and json was 15 kb. If it helps anyone.

[JL Peyret]

Wrt to pickle, it's a pretty sad state of affairs that the word security does not appear anywhere in this page. security.stackexchange.com/questions/183966/…

[baxx]

why is this easier than json.dump( ) as outlined in the other answer?

[ansh sachdeva]

note : You cannot use the json method for saving dictionary object multiple times, since w is used for opening the file and it will overwrite the contents with new content every time those lines are executed (same goes for pickle i guess, didn't really tried that one)

[ansh sachdeva]

@VineeshTP maybe this will help

[Vineesh TP]

@anshsachdeva JSON.stringify(result_dict, null, 2); '2' worked for me.

[bit_scientist]

Does this package do all what json does? I mean is it fully replaceable with json all the time?