Pickling multiple dictionaries

Question

So I'm new to python and I'll most likely be asking many noob questions throughout my learning process (If they haven't already been asked/answered of course).

One question I have is if there is a way to save multiple dictionaries to one text file using pickle, or if each individual dictionary has to be saved to it's own separate file. For example, if I want to create a program to manage web accounts, with each account having a variety of arbitrary keys/values, can I save all these individual accounts to one archive as separate dictionaries?

Thanks in advance, and a noob would appreciate example code and/or any suggestions.

Solution 1

One quick solution is to place both dictionaries in a list, then pickle the list.

Here is an example that writes it out as a binary file.

    import cPickle as pkl

    MyFirstDict = { "hat": 7, "carpet": 5 }
    MySecondDict = { "syrup":15, "yogurt": 18 }

    MyDicts = [MyFirstDict, MySecondDict]

    pkl.dump( MyDicts, open( "myDicts.p", "wb" ) )

Here is one that loads it.

    import cPickle as pkl

    myDicts = pkl.load( open ("myDicts.p", "rb") )

If you need the file to be human readable, consider writing it out as a regular text file instead. Be warned, this is far less efficient and leaves the data especially exposed for something like web account data.

    MyFirstDict = { "hat": 7, "carpet": 5 }
    MySecondDict = { "syrup":15, "yogurt": 18 }

    MyDicts = [MyFirstDict, MySecondDict]

    outputFile = open( "myDicts.txt", "w")
    outputFile.write(str(MyDicts))
    outputFile.flush()
    outputFile.close()

And to read it back in...

    import ast

    inputFile = open( "myDicts.txt", "r")
    lines = inputFile.readlines()

    objects = []
    for line in lines:
        objects.append( ast.literal_eval(line) )

    myDicts = objects[0]

References:

// reference for pickle http://wiki.python.org/moin/UsingPickle

// source for text-to-object solution Python convert string object into dictionary

Solution 2

There are a few ways of pickling multiple objects. Which is best will depend a bit on your use case.

One option is to put all your separate account dictionaries into a larger data structure, such as a list or dictionary (perhaps keyed by account ID). Then you pickle the larger data structure and the account data will be saved in one go. Note that Pickle's format is binary, not text. If you want something human readable you should probably use json or yaml instead.

Another option is to write several pickle values to a file in sequence. This works pretty much as you'd expect, just call pickle.dump repeatedly to save them, and pickle.load repeatedly to load them back in. One downside to this approach is that you can't easily search for a specific data item that you need, you need to keep loading them in sequence until you find the one you want. Another is that the identity of objects that are pickled more than once will be lost (the values will probably be equal when you load them, but not references to the same object). An advantage is that you can append more data to the end of the file without reading in the previous stuff, just to write it out again.

A third option is to use the shelve module from the standard library. It acts somewhat like a dictionary, but the keys and values are pickled to a file. It may solve the lookup issue of the separate pickles above, but it doesn't fix the loss of object identity between values.

A final idea is to not use pickle at all, but instead use a real database system. This may be a bit harder to code, but may let you avoid issues of concurrency and data integrity that may otherwise be difficult to solve.