How can I pickle a python object into a csv file?

Question

I am trying to pickle a python object into a csv file. I want to write the pickle of an object as the third column in my file. I want to use pickle to avoid writing serialization for my complex objects.

Code to write to csv :

 with open(self.file_path, 'a') as csv_file:
        wr = csv.writer(csv_file, delimiter='|')
        row = ['klines', symbol]
        row.extend(pickle.dumps(object))
        wr.writerow(row)

Code to read csv :

 with open(self.simulation_file_name, 'r') as csv_file:
        line = csv_file.readline()
        while line != '':
            line = line.strip('\n')
            columns = line.split('|')
            event_type = line.pop(0)
            symbol = line.pop(0)
            pickled = line.pop(0)
            klines = pickle.loads(klines)

I get the following error :

TypeError: a bytes-like object is required, not 'str'

Solution

To write bytes/binary in text file like CSV, use base64 or other methods to avoid any escaping problem. Code simplified & python3 assumed.

import base64
with open('a.csv', 'a', encoding='utf8') as csv_file:
    wr = csv.writer(csv_file, delimiter='|')
    pickle_bytes = pickle.dumps(obj)            # unsafe to write
    b64_bytes = base64.b64encode(pickle_bytes)  # safe to write but still bytes
    b64_str = b64_bytes.decode('utf8')          # safe and in utf8
    wr.writerow(['col1', 'col2', b64_str])


# the file contains
# col1|col2|gANdcQAu

with open('a.csv', 'r') as csv_file:
    for line in csv_file:
        line = line.strip('\n')
        b64_str = line.split('|')[2]                    # take the pickled obj
        obj = pickle.loads(base64.b64decode(b64_str))   # retrieve

P.S. If you are not writing a utf8 file (e.g. ascii file), simply replace the encoding method.

P.S. Writing bytes in CSV is possible yet hardly elegant. One alternative is dumping a whole dict with dumped objects as values and storing keys in the CSV.

Comments

[constt]

Wouldn't it be better to store in a csv file not a pickled python object itself but a link to it saved somewhere on a disk?

[DYZ]

CSV files are designed to be human-readable, but Pickle is not.

[Blownhither Ma]

pickle.dumps gives a bytes object. list.extend with a bytes object extends the list with int's, resulting in ['klines', symbol, 128, 3, 93...] (tested in Python3). Is this what you want? Or you mean row.append

[Taku]

A full traceback would be nice

[user1767754]

Can you tell as well why you want to do this to not have a x/y problem?