Python base64 encoding a list

Question

Encoding is new to me in Python, and I am trying to understand it. Apologies if this has been asked and answered already.

I am trying to encode a Python list and decode it. When I am trying to encode a list directly, I am hitting an error like below.

>>> my_list = [1, 2, 3]
>>> encoded_list = base64.b64encode(my_list)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 54, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: b2a_base64() argument 1 must be string or buffer, not list

To fix it, I converted the list object to a string and passed it to the encode function and I was able to successfully encode it.

>>> encoded_list = base64.b64encode(str(my_list))
>>> encoded_list
'WzEsIDIsIDNd'

When I try to decode it, I get a decoded string like below.

>>> decoded_list = base64.b64decode(encoded_list)
>>> decoded_list
'[1, 2, 3]'
>>> type(decoded_list)
<type 'str'>

But my original intention was to encode and decode a list and not convert the list to a string and then string to list.

Pretty sure this is not the right way to encode objects like dict or a list. If that's the case, Can someone please enlighten me on how to encode/decode non string objects in Python?

Thanks very much.

Solution 1

Try encoding/decoding using JSON instead of string.

import json
import base64

my_list = [1, 2, 3]
json_encoded_list = json.dumps(my_list)
#: '[1, 2, 3]'
b64_encoded_list = base64.b64encode(json_encoded_list)
#: 'WzEsIDIsIDNd'
decoded_list = base64.b64decode(b64_encoded_list)
#: '[1, 2, 3]'
my_list_again = json.loads(decoded_list)
#: [1, 2, 3]

But in practice, for pretty much any storage reasons I can think of there's no real reason to base64 encode your json output. Just encode and decode to json.

my_list = [1, 2, 3]
json_encoded_list = json.dumps(my_list)
#: '[1, 2, 3]'
my_list_again = json.loads(json_encoded_list)
#: [1, 2, 3]

If you need anything more complicated than Arrays and Dictionaries, then probably go with 7stud's pickle method. However JSON is simple, readable, widely supported and cross-compatible with other languages. I'd choose it whenever possible.

Solution 2

You are interested in the data being encoded not the list itself being encoded. Therefore I suggest the following: use struct to pack the data.

x = range(10)
import struct
y = struct.pack('<{}i'.format(len(x)), *x)
import base64
z = base64.b64encode(y)

z will now be an encoding of the data in the list.

You can decode it back and retrieve the list as follows:

y = base64.b64decode(z)
list(struct.unpack('<{}i'.format(len(y)/4), y))

Comments

[Gareth Ma]

I am not a pro at this, but try to do this: map(base64.b64encode, my_list)

[Gareth Ma]

That will encode each of the element of the list

[Gareth Ma]

and to reverse, do map(base64.b64decode, my_list)

[DYZ]

What is the purpose of this exercise? (The answer may depend on what you plan to do with the encoded string.)

[bipll]

You don't encode Python objects, only strings. Did you mean string '123' or '\1\2\3'? The first is ''.join(map(str, my_list)) and the second ''.join(map(chr, my_list)).

[Joel Cornett]

@DyZ no. JSON scalars include lowercase boolean values, which are not valid in Python, and “null” which again is not found in Python. Finally, python string representations are rendered with single quotes. The JSON spec accepts only double quotes. I could go on, but the point is that while JSON and python data structures look similar, they are not directly interchangeable

[DYZ]

@JoelCornett True, but for the purpose of this question (a list of numbers) the difference is irrelevant.

[Joel Cornett]

For basic data structures, don’t forget about ast.literal_eval!