TypeError: ObjectId('') is not JSON serializable

Question

My response back from MongoDB after querying an aggregated function on document using Python, It returns valid response and i can print it but can not return it.

Error:

TypeError: ObjectId('51948e86c25f4b1d1c0d303c') is not JSON serializable

Print:

{'result': [{'_id': ObjectId('51948e86c25f4b1d1c0d303c'), 'api_calls_with_key': 4, 'api_calls_per_day': 0.375, 'api_calls_total': 6, 'api_calls_without_key': 2}], 'ok': 1.0}

But When i try to return:

TypeError: ObjectId('51948e86c25f4b1d1c0d303c') is not JSON serializable

It is RESTfull call:

@appv1.route('/v1/analytics')
def get_api_analytics():
    # get handle to collections in MongoDB
    statistics = sldb.statistics

    objectid = ObjectId("51948e86c25f4b1d1c0d303c")

    analytics = statistics.aggregate([
    {'$match': {'owner': objectid}},
    {'$project': {'owner': "$owner",
    'api_calls_with_key': {'$cond': [{'$eq': ["$apikey", None]}, 0, 1]},
    'api_calls_without_key': {'$cond': [{'$ne': ["$apikey", None]}, 0, 1]}
    }},
    {'$group': {'_id': "$owner",
    'api_calls_with_key': {'$sum': "$api_calls_with_key"},
    'api_calls_without_key': {'$sum': "$api_calls_without_key"}
    }},
    {'$project': {'api_calls_with_key': "$api_calls_with_key",
    'api_calls_without_key': "$api_calls_without_key",
    'api_calls_total': {'$add': ["$api_calls_with_key", "$api_calls_without_key"]},
    'api_calls_per_day': {'$divide': [{'$add': ["$api_calls_with_key", "$api_calls_without_key"]}, {'$dayOfMonth': datetime.now()}]},
    }}
    ])


    print(analytics)

    return analytics

db is well connected and collection is there too and I got back valid expected result but when i try to return it gives me Json error. Any idea how to convert the response back into JSON. Thanks

Solution 1

You should define you own JSONEncoder and using it:

import json
from bson import ObjectId

class JSONEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, ObjectId):
            return str(o)
        return json.JSONEncoder.default(self, o)

JSONEncoder().encode(analytics)

It's also possible to use it in the following way.

json.encode(analytics, cls=JSONEncoder)

Solution 2

Pymongo provides json_util - you can use that one instead to handle BSON types

def parse_json(data):
    return json.loads(json_util.dumps(data))

Solution 3

>>> from bson import Binary, Code
>>> from bson.json_util import dumps
>>> dumps([{'foo': [1, 2]},
...        {'bar': {'hello': 'world'}},
...        {'code': Code("function x() { return 1; }")},
...        {'bin': Binary("")}])
'[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$code": "function x() { return 1; }", "$scope": {}}}, {"bin": {"$binary": "AQIDBA==", "$type": "00"}}]'

Actual example from json_util.

Unlike Flask's jsonify, "dumps" will return a string, so it cannot be used as a 1:1 replacement of Flask's jsonify.

But this question shows that we can serialize using json_util.dumps(), convert back to dict using json.loads() and finally call Flask's jsonify on it.

Example (derived from previous question's answer):

from bson import json_util, ObjectId
import json

#Lets create some dummy document to prove it will work
page = {'foo': ObjectId(), 'bar': [ObjectId(), ObjectId()]}

#Dump loaded BSON to valid JSON string and reload it as dict
page_sanitized = json.loads(json_util.dumps(page))
return page_sanitized

This solution will convert ObjectId and others (ie Binary, Code, etc) to a string equivalent such as "$oid."

JSON output would look like this:

{
  "_id": {
    "$oid": "abc123"
  }
}

Solution 4

Most users who receive the "not JSON serializable" error simply need to specify default=str when using json.dumps. For example:

json.dumps(my_obj, default=str)

This will force a conversion to str, preventing the error. Of course then look at the generated output to confirm that it is what you need.

Solution 5

from bson import json_util
import json

@app.route('/')
def index():
    for _ in "collection_name".find():
        return json.dumps(i, indent=4, default=json_util.default)

This is the sample example for converting BSON into JSON object. You can try this.

View more solutions

Comments

[Irfan]

Perfect! It worked for me. I already have a Json encoder class, How can i merge that with yours class?My already Json encode class is: 'class MyJsonEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, datetime): return str(obj.strftime("%Y-%m-%d %H:%M:%S")) return json.JSONEncoder.default(self, obj)'

[defuz]

@IrfanDayan, just add if isinstance(o, ObjectId): return str(o) before return in method default.

[Joshua Powell]

I agree with @tim, this is correct way to deal with BSON data coming from mongo. api.mongodb.org/python/current/api/bson/json_util.html

[jonprasetyo]

Yes, seems to be more of a hassle free if we use this way

[oferei]

Just to clarify, no need to call 'jsonify' directly from a Flask request handler - just return the sanitized result.

[Garren S]

You're absolutely right. A Python dict (which json.loads returns) should automatically be jsonified by Flask.

[Liviu Chircu]

Could you add from bson import ObjectId, so everybody can copy-paste even faster? Thanks!

[Rahul]

That's the best way actually.

[Jake]

An example here would be a little more helpful, as this is the best way but the linked documentation isn't the most user friendly for noobs

[SouvikMaji]

Isn't a dict object not callable?

[Garren S]

@rick112358 how does a dict not being callable relate to this Q&A?

[Ashish]

you can also use json_util.loads() to get the exact same dictionary back (instead of one with '$oid' key).

[Kevin]

@defuz Why not just use str? What's wrong with that approach?

[Varij Kapil]

@defuz: When I try to use this, ObjectID is removed, but my json response is broken into single characters. I mean when I print each element from the resulting json in a for loop I get each character as an element. Any idea how to solve this?

[Muhriddin Ismoilov]

in this case you are not passing '_id' attribute , instead just deleted '_id' and passed other attributes of doc

[tsveti_iko]

this is very useful for the pytest-mongodb plugin when creating fixtures

[Vexy]

+1 Ha ! Could it have been more simpler 😍 Generally speaking; to avoid all the fuzz with custom encoders and bson importing, cast ObjectID to string: object['_id'] = str(object['_id'])

[nbs]

from bson import json_util json.loads(json_util.dumps(user_collection)) ^ this worked after installing python-bsonjs with pipenv install python-bsonjs

[Hans Ginzel]

return json.JSONEncoder.default(self, o) better written as return supper().default(self, o)