Unable to deserialize PyMongo ObjectId from JSON

Question

I'm seemingly unable to deserialize my MongoDB JSON document with the BSON json_util.

The json.loads function is choking on the ObjectId() string. I had understood json_util capable of handling MongoDB's ObjectId format and transforming into usable JSON.

Python code:

import json    
from bson import json_util

s = "{u'_id': ObjectId('4ed559abf047050c58000000')}"
u = json.loads(s, object_hook=json_util.object_hook)

I get the decoder exception:

...
    u = json.loads(s, object_hook=json_util.object_hook)
  File "\python27\lib\json\__init__.py", line 339, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "\python27\lib\json\decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "\python27\lib\json\decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 1 (char 1)

Am I missing something?

Solution 1

I think your string form actually looks like the python representation...

s = '{"_id": {"$oid": "4edebd262ae5e93b41000000"}}'
u = json.loads(s, object_hook=json_util.object_hook)

print u  # Result:  {u'_id': ObjectId('4edebd262ae5e93b41000000')}

s = json.dumps(u, default=json_util.default)

print s  # Result:  {"_id": {"$oid": "4edebd262ae5e93b41000000"}}

The bson.json_util.object_hook function does not seem to have any type of handling for there being ObjectId() in the actual json string representation.

Solution 2

There are two problems here:

1. The string you're attempting to JSON-decode is not JSON, it's the string representation of a Python dictionary. In particular, the problem is that u'_id' is not a valid JSON key (JSON keys are quoted strings; the "u" here indicates a Python unicode string, which is meaningless in JSON)

2. json_util.object_hook doesn't make ObjectId available to JSON; the json module will decode the JSON, and then call the object_hook callback with each decoded object. json_util.object_hook will look for certain patterns as defined in the strict mode of MongoDB Extended JSON.

See @jdi's answer for examples of how to properly use json_util.

Comments

[nathanmgroom]

RE: 1. Hi dcrosta, the string I had used in my example was taken from the app log, hence the extra python unicode 'u'.

[nathanmgroom]

Ok, after looking at @dcrosta's suggestion to review MongoDB Extended JSON mongodb.org/display/DOCS/Mongo+Extended+JSON, let me instead try converting the data_oid from TenGen BSON type to Strict JSON, i.e. "$oid".

[nathanmgroom]

The json.dumps suggestion worked perfectly. s = json.dumps(u, default=json_util.default), output {"_id": {"$oid": "4ed559abf047050c58000000"}}. Thanks @jdi

[Sarada Akurathi]

Hi All, I also want to conver ObjectId() of the mongo result, i tried to execute above code, i am getting the following error NameError: global name 'json_util' is not defined is any library i need to import? , i have imported bson library but still same error

[jdi]

@SaradaAkurathi when I wrote this answer over. 5 years ago, json_util was part of the bson module (as documented in the answer). Is that no longer the case?

[Sarada Akurathi]

@jdi, i found this json_util for json_util. option bson.json_util.default is still there. problem earlier with wrong import, now i imported correct library, the above error is gone but still result is not converted. Can you help what went wrong

[jdi]

@SaradaAkurathi I am not sure what your issue is. You may be best served by opening a new question with the details of your problem. This question is quite old, and comments are not really the best place to troubleshoot

[Ashwini]

object_hook=json_util.object_hook does the magic. Thanks for sharing:)