json.dump throwing "TypeError: {...} is not JSON serializable" on seemingly valid object?

Question

Background: I am writing a python program which should manage my music files. It crawls directories and puts the files and their meta data (via mutagen), encoded in JSON, in a file as a simple "database". I have the directory searching fine, but when I try and save the database, or encode to JSON, it throws a "TypeError: {...} is not JSON serializable" (the ... are some keys and values from a dict, more on that below)

The Problem: The program builds a large dictionary object following this format:

{
    "<song id>":{
        "artist":"<song artist>",
        "album":"<song album>",
        "title":"<song title>"},
    ...
}

Every single song file is indexed via this format. When I try to dump the database to a file, I get this:

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    sit()
  File "D:\workbench\ideas\musicmanager\v0\spider.py", line 116, in sit
    json.dump(js.db,f,True)
  File "C:\Python27\lib\json\__init__.py", line 181, in dump
    for chunk in iterable:
  File "C:\Python27\lib\json\encoder.py", line 428, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "C:\Python27\lib\json\encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "C:\Python27\lib\json\encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "C:\Python27\lib\json\encoder.py", line 436, in _iterencode
    o = _default(o)
  File "C:\Python27\lib\json\encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: {'album': [u"Rooney's Lost Album"], 'title': [u'The Kids
After Sunset'], 'artist': [u'Rooney']} is not JSON serializable

With the key for that particular song entry being

Rooney|Rooney's Lost Album|The Kids After Sunset|The Kids After Sunset.itunes.mp3

(the format for the id is a little bulky, I might end up hashing that...)

So I tried to

json.dumps({'album': [u"Rooney's Lost Album"], 'title': [u'The Kids
After Sunset'], 'artist': [u'Rooney']})

which worked fine, as did

json.dumps({"Rooney|Rooney's Lost Album|The Kids After Sunset|The Kids
After Sunset.itunes.mp3":""})

And then I tried this:

rooney = "Rooney|Rooney's Lost Album|The Kids After Sunset|The Kids
    After Sunset.itunes.mp3"
json.dumps({rooney:js.db['songsbyid'][rooney]})

Which failed with the type error again.

Why does that object fail with json.dump? I have plenty of other objects with keys containing pipes "|" and apostrophes "'"... At the moment, I have no way for anyone else to test this, should I post a pickled version of the database object?

Additional Notes

• The resulting object below json.dumps is fine, so I am wondering if the issue has to do with the size of the database in any way?

  {rooney:js.db['songsbyid'][rooney]} {"Rooney|Rooney's Lost Album|The Kids After Sunset|The Kids After Sunset.itunes.mp3": {'album': [u"Rooney's Lost Album"], 'title': [u'The Kids After Sunset'], 'artist': [u'Rooney']}}

• If I exclude the song by renaming the extension so the script ignores it, another arbitrary song causes the same error. I renamed&excluded this new song, and ran into ANOTHER new song... I don't know how many there are.

• I changed my program to crawl the next furthest sub-directory containing the original problem song, and json.dump raised a TypeError on a completely different song...

Solution 1

Because it's not actually a dictionary; it's another mapping type that looks like a dictionary. Use type() to verify. Pass it to dict() to get a real dictionary from it.

Solution 2

In my case, boolean values in my Python dict were the problem. JSON boolean values are in lowercase ("true", "false") whereas in Python they are in Uppercase ("True", "False"). Couldn't find this solution anywhere online but hope it helps.

Solution 3

I wrote a class to normalize the data in my dictionary. The 'element' in the NormalizeData class below, needs to be of dict type. And you need to replace in the __iterate() with either your custom class object or any other object type that you would like to normalize.

class NormalizeData:

    def __init__(self, element):
        self.element = element

    def execute(self):
        if isinstance(self.element, dict):
            self.__iterate()
        else:
            return

    def __iterate(self):
        for key in self.element:
            if isinstance(self.element[key], <ClassName>):
                self.element[key] = str(self.element[key])

            node = NormalizeData(self.element[key])
            node.execute()

Comments

[drdrez]

I tried to use dict() around the object in both the json.dump of the entire database object, and the "{rooney:js.db['songsbyid'][rooney]}" snippet (the first culprit song entry) and I am still getting a type error... :/

[drdrez]

The issue was deeper in my code- I was using mutagen to get the meta data for the music files, and the values it returns must not be actual lists. Using dict() on the values it returned fixed the issue.

[Jessica Pennell]

I'm posting this for the benefit of other googlers that get frustrated trying this advice and feel a bit like Patrick Star talking to Man Ray when they assert a ton of ways something really is a dict. If anything inside your dictionary isn't serializable, you'll get this typeerror even if you are serializing a dictionary at the top level. Try print(str(yourdict)) and compare the output of that to your error output to see which object.