List of objects to JSON with Python

Question

I have a problem converting Object instances to JSON:

ob = Object()

list_name = scaping_myObj(base_url, u, number_page)

for ob in list_name:
   json_string = json.dumps(ob.__dict__)
   print json_string

In list_name I have a list of Object instances.

json_string return, for example:

{"city": "rouen", "name": "1, 2, 3 Soleil"}
{"city": "rouen", "name": "Maman, les p'tits bateaux"}

But I would like just 1 JSON string with all the info in a list:

[{"city": "rouen", "name": "1, 2, 3 Soleil"}, {"city": "rouen", "name": "Maman, les p'tits bateaux"}]

Solution 1

You can use a list comprehension to produce a list of dictionaries, then convert that:

json_string = json.dumps([ob.__dict__ for ob in list_name])

or use a default function; json.dumps() will call it for anything it cannot serialise:

def obj_dict(obj):
    return obj.__dict__

json_string = json.dumps(list_name, default=obj_dict)

The latter works for objects inserted at any level of the structure, not just in lists.

Personally, I'd use a project like marshmallow to handle anything more complex; e.g. handling your example data could be done with

from marshmallow import Schema, fields

class ObjectSchema(Schema):
    city = fields.Str()
    name = fields.Str()

object_schema = ObjectSchema()
json_string = object_schema.dumps(list_name, many=True)

Solution 2

Similar to @MartijnPieters' answer, you can use the json.dumps default parameter with a lambda, if you don't want to have to create a separate function: json.dumps(obj, default = lambda x: x.__dict__)

Solution 3

Another possible solution to this problem is jsonpickle which can be used to transform any Python object into JSON (not just simple lists).

From the jsonpickle home page:

jsonpickle is a Python library for serialization and deserialization of complex Python objects to and from JSON. The standard Python libraries for encoding Python into JSON, such as the stdlib’s json, simplejson, and demjson, can only handle Python primitives that have a direct JSON equivalent (e.g. dicts, lists, strings, ints, etc.). jsonpickle builds on top of these libraries and allows more complex data structures to be serialized to JSON. jsonpickle is highly configurable and extendable–allowing the user to choose the JSON backend and add additional backends.

Performing a transformation is simple:

import jsonpickle

class JsonTransformer(object):
    def transform(self, myObject):
        return jsonpickle.encode(myObject, unpicklable=False)

Comments

[Martijn Pieters]

You mean you want a JSON list?

[pedro]

Here I have 2 Json so I want only one with all the info like [{"city": "rouen", "name": "1, 2, 3 Soleil"}, {"city": "rouen", "name": "Maman, les p'tits bateaux"}]

[pedro]

your first solution works perfectly. Thanks a lot :)

[Matthias]

That looks like a great solution. At the moment I'm trying to jsonify a complex data structure and this could be the way to go.

[Roon13]

I haven't tried second solution Because your first solution worked for me :)

[SkyLeach]

It should be noted that deserialization in this way becomes tricky (subnested objects will be dicts on json.loads.

[Martijn Pieters]

@SkyLeach: yes; one convention is to add an extra key to the dictionary with the object type, (e.g. `_type': 'ObjectType') then on deserialisation walk the dictionaries to detect those. Still somewhat painful.

[SkyLeach]

@MartijnPieters exactly. I use the kwarg fromjson in my base class and then child classes are documented to traverse any subnested objects like so: if 'fromsjson' in kwargs: subclass = SubClass(fromjson=json.loads(self.subclass); etc...

[penchant]

this is great and all, but it requires instructing anyone else who uses your objects on your magical way to get json output. How do you make it to where json.dumps([obj1, obj2, obj3, obj4]) works? this way anyone who uses your library can json.dumps as usual without having to go look at a bunch of readmes

[Martijn Pieters]

@tr3buchet: there is no such way. JSON is not a format that can accommodate all object types.

[penchant]

@MartijnPieters I understand JSON can't accommodate all object types, I'm trying to figure a way to make my object as such that it can be accommodated by JSON with zero configuring. An example of what I mean is that I can subclass dict and suddenly my custom object is serializable.

[Martijn Pieters]

@tr3buchet: and that's no less magical, and won't include attributes of your custom object, only the key-value pairs. There is no silver bullet here.

[penchant]

That only matters if I am serializing things like with pickle. A pickled object is of little value unless you deserialize it later. This is not at all true with json. The purpose of json isn't so much serialization and deserialization, but more for outputting a machine readable format that others can consume as they see fit (probably not even in Python). Why do we need custom encoders for every non-builtin type imaginable? What's wrong with a __json__ function? Why force everyone to subclass things that they should not be subclassing (dict) or monkey patch json just to be able to do this?

[Martijn Pieters]

@tr3buchet: Because there is no standard serialisation. Python can't just invent one there.

[penchant]

@MartijnPieters correct! I wouldn't expect python to know what the json representation of my objects should be. I would expect python to give me a way to define that representation in my class definitions such that json would work out of the box with my objects without requiring these custom encoder shenanigans. It's painful because I can't make a library that works with json.dumps(), I have to create an encoder and users have to go find that encoder and use it anytime they want to output json.