How do I read binary pickle data first, then unpickle it?

Question

I'm unpickling a NetworkX object that's about 1GB in size on disk. Although I saved it in the binary format (using protocol 2), it is taking a very long time to unpickle this file---at least half an hour. The system I'm running on has plenty of system memory (128 GB), so that's not the bottleneck.

I've read here that pickling can be sped up by first reading the entire file into memory, and then unpickling it (that particular thread refers to python 3.0, which I'm not using, but the point should still be true in python 2.6).

How do I first read the binary file, and then unpickle it? I have tried:

import cPickle as pickle
f = open("big_networkx_graph.pickle","rb")
bin_data = f.read()
graph_data = pickle.load(bin_data)

But this returns:

TypeError: argument must have 'read' and 'readline' attributes

Any ideas?

Solution 1

pickle.load(file) expects a file-like object. Instead, use:

pickle.loads(string)

Read a pickled object hierarchy from a string. Characters in the string past the pickled object’s representation are ignored.

Solution 2

The documentation mentions StringIO, which I think is one possible solution.

Try:

f = open("big_networkx_graph.pickle","rb")
bin_data = f.read()
sio = StringIO(bin_data)
graph_data = pickle.load(sio)