Can json.loads ignore trailing commas?

Question

As mentioned in this StackOverflow question, you are not allowed to have any trailing commas in json. For example, this

{
    "key1": "value1",
    "key2": "value2"
}

is fine, but this

{
    "key1": "value1",
    "key2": "value2",
}

is invalid syntax.

For reasons mentioned in this other StackOverflow question, using a trailing comma is legal (and perhaps encouraged?) in Python code. I am working with both Python and JSON, so I would love to be able to be consistent across both types of files. Is there a way to have json.loads ignore trailing commas?

Solution 1

You can wrap python's json parser with jsoncomment

JSON Comment allows to parse JSON files or strings with:

• Single and Multi line comments
• Multi line data strings
• Trailing commas in objects and arrays, after the last item

Example usage:

import json
from jsoncomment import JsonComment

with open(filename) as data_file:    
    parser = JsonComment(json)
    data = parser.load(data_file)

Solution 2

Strip the commas before you pass the value in.

import re

def clean_json(string):
    string = re.sub(",[ \t\r\n]+}", "}", string)
    string = re.sub(",[ \t\r\n]+\]", "]", string)

    return string

Solution 3

In python you can have trailing commas inside of dictionaries and lists, so we should be able to take advantage of this using ast.literal_eval:

import ast, json

str = '{"key1": "value1", "key2": "value2",}'

python_obj = ast.literal_eval(str) 
# python_obj is {'key1': 'value1', 'key2': 'value2'}

json_str = json.dumps(python_obj)
# json_str is '{"key1": "value1", "key2": "value2"}'

However, JSON isn't exactly python so there are a few edge cases to this. For example, values like null, true, false don't exist in python. We can replace those with valid python equivalents before we run the eval:

import ast, json

def clean_json(str):
  str = str.replace('null', 'None').replace('true', 'True').replace('false', 'False')
  return json.dumps(ast.literal_eval(str))

This will unfortunately mangle any strings that have the words null, true, or false in them.

{"sentence": "show your true colors"} 

would become

{"sentence": "show your True colors"}

Solution 4

Fast forward to 2021, now we have https://pypi.org/project/json5/

A quote from the link:

A Python implementation of the JSON5 data format.

JSON5 extends the JSON data interchange format to make it slightly more usable as a configuration language:

• JavaScript-style comments (both single and multi-line) are legal.
• Object keys may be unquoted if they are legal ECMAScript identifiers
• Objects and arrays may end with trailing commas.
• Strings can be single-quoted, and multi-line string literals are allowed.

Usage is consistent with python's built in json module:

>>> import json5
>>> json5.loads('{"key1": "{my special value,}",}')
{u'key1': u'{my special value,}'}

It does come with a warning:

Known issues

• Did I mention that it is SLOW?

It is fast enough for loading start up config etc.

Solution 5

Cobbling together the knowledge from a few other answers, especially the idea of using literal_eval from @Porkbutts answer, I present a wildly-evil solution to this problem

def json_cleaner_loader(path):
    with open(path) as fh:
        exec("null=None;true=True;false=False;d={}".format(fh.read()))
    return locals()["d"]

This works by defining the missing constants to be their Pythonic values before evaluating the JSON struct as Python code. The structure can then be accessed from locals() (which is yet another dictionary).

This should work with both Python 2.7 and Python 3.x

BEWARE this will execute whatever is in the passed file, which may do anything the Python interpreter can, so it should only ever be used on inputs which are known to be safe (ie. don't let web clients provide the content) and probably not in any production environment.
This probably also fails if it's given a very large amount of content.

View more solutions

Comments

[user2357112]

This might look okay, but it'll mangle inputs like '{"foo": ",}"}'.

[Rob Watts]

Technically there would need to be a space in there, but yes, ", }" would get mangled.

[Sven]

That package isn't very good. It removes commas from strings as well. Just have a string containing ,} or ,] and the commas will magically disappear.

[Sven]

Why have such a complicated regexp? Just use ',\s*}' and ',\s*]'. Then again, this method mangles strings wnich is bad. The JsonComment package isn't much better as it uses a method similar to this.

[jpmc26]

No. You do not parse formats that allow nested elements using regular expressions. -1

[Cramer]

@jpmc26 They're not parsing the whole tree, just a single piece of syntax which itself has no nesting. The issue is not the nesting but the matching of the quotes to determine they're not in a string which is possible, but painful.

[jpmc26]

@Cramer It fails on this JSON: {"key1": "{my special value, }"} (tio demo). You do not make assumptions about the contents of complex formats like JSON. It's always a bad idea. Just don't do it. Use a properly tested parser and save yourself the heartburn.

[jpmc26]

Also, string is a standard module. s would be a better variable name.

[jpmc26]

As @Sven says, here's a test string to demo it: {"key1": "{my special value,}"}.

[Cramer]

You didn't read my comment did you? It's the being-inside-a-quote that's the issue. Is there a situation that breaks that is NOT inside a string?

[rrauenza]

@Sven Looks like they upgraded to a proper parse and abandoned regex: github.com/vaidik/commentjson/releases

[Rob Watts]

This is a bad answer that should never be used, but technically will work. +1 only because you did include a disclaimer.

[ti7]

Thanks - I'm quite proud of how horrible it is!

[Arthur Tacca]

If it has its own full-blown JSON parser why not just return the result of its parsing? This is parsing it, re-encoding it then parsing it again.

[jarmod]

Mangling of the words null, true, or false in a string is one heck of a downside, so thanks for highlighting it.