bytearray is not JSON serializable

Question

I get this error message in Python/Django: bytearray(b'') is not JSON serializable. Currently, I do it like this:

...
from django.core.serializers.json import DjangoJSONEncoder
...

output = json.dumps(data, cls=DjangoJSONEncoder)
return HttpResponse(output)

I import DjangoJSONEncoder, because before that I had problem with datetime. That problem went away, but now I have another problem with type bytearray. So, I'm interested to know, if there is a standard and universal way of creating json string, no matter what data types you use.

Solution 1

For types that json cannot encode, you can use cPickle.dumps() to convert to a string type first and then use json.

Keep in mind this warning from the docs before using cPickle though:

Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Solution 2

As @Martijn Pieters suggested in his comment, Json encoder does not know how to convert a bytearray into the corresponding json notation.

check out https://docs.python.org/2/library/json.html#encoders-and-decoders for the datatypes that json understand. DjangoJSONEncoder uses the same subset, with an understanding of django querysets.

A solution to your problem could be to implement a custom encoder for bytearray, and handle the response properly in the UI.

Comments

[Martijn Pieters]

No, of course there isn't. How would you serialise an arbitrary data type without a-priori knowledge of what it models? Just because you as a programmer understand what a bytearray represents doesn't mean the JSON encoder would know.

[Yann Vernier]

In particular, for bytearray it wouldn't even be obvious whether to serialize it to a string or list (JSON doesn't have all that many types to choose from).

[Jacobian]

I examined table in database and found out, that this is because of a field of bit type, which takes only 0 or 1 value. So, it is rather strange that such a standard and "simple" type can not be handled in Python/Django

[Yann Vernier]

It is most certainly not a Python limitation! JSON is a restricted type set by design (for interoperability and safety); don't confuse it with general serialization like pickling.

[Martijn Pieters]

@Jacobian: why is this surprising? Just because there is a possible mapping doesn't mean that it is the correct mapping in all use cases. Mapping a byte array with only 0s and 1s could be mapped to a list of booleans, or to a string, or to a list of integers, and all 3 could represent the information in JSON. But what if those bytes represent characters? Or small integers?

[Jacobian]

Thanks, guys! Seems, as if there is a solution - jsonpickle library

[Yann Vernier]

Allow me to quote the big red box at the top of the pickle documentation: "Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source."

[Jacobian]

@ Yann Vernier. It is an important point. Fortunately, in my project I will deal with a trusted database