Python: Comparing two JSON objects in pytest

Question

I have an API that returns this JSON response

{
    "message": "Staff name and password pair not match",
    "errors": {
        "resource": "Login",
        "field": "staff_authentication",
        "code": "invalid",
        "stack_trace": null
    }
}

Using pytest, I want to build a copy of the JSON object and make sure it is exactly the same

import pytest
import json
from collections import namedtuple
from flask import url_for
from myapp import create_app

@pytest.mark.usefixtures('client_class')
class TestAuth:

    def test_login(self, client):
        assert client.get(url_for('stafflogin')).status_code == 405
        res = self._login(client, 'no_such_user', '123456')
        assert res.status_code == 422
        response_object = self._json2obj(res.data)
        assert response_object.message == 'Staff name and password pair not match'
        invalid_password_json = dict(message="Staff name and password pair not match",
                                    errors=dict(
                                        resource="Login",
                                        code="invalid",
                                        field="staff_authentication",
                                        stack_trace=None,)
                                    )
        assert self._ordered(response_object) == self._ordered(invalid_password_json)

    def _login(self, client, staff_name, staff_password):
        return client.post('/login',
            data=json.dumps(dict(staff_name=staff_name, staff_password=staff_password)),
            content_type='application/json',
            follow_redirects=True)

    def _json_object_hook(self, d): return namedtuple('X', d.keys())(*d.values())
    def _json2obj(self, data): return json.loads(data, object_hook=self._json_object_hook)

    def _ordered(self, obj):
        if isinstance(obj, dict):
            return sorted((k, self._ordered(v)) for k, v in obj.items())
        if isinstance(obj, list):
            return sorted(self._ordered(x) for x in obj)
        else:
            return obj

pytest shows that the 2 objects are unequal.

>       assert self._ordered(response_object) == self._ordered(invalid_password_json)
E       AssertionError: assert X(message='St...k_trace=None)) == [('errors', [(...r not match')]
E         At index 0 diff: 'Staff name and password pair not match' != ('errors', [('code', 'invalid'), ('field', 'staff_authentication'), ('resource', 'Login'), ('stack_trace', None)])
E         Full diff:
E         - X(message='Staff name and password pair not match', errors=X(resource='Login', field='staff_authentication', code='invalid', stack_trace=None))
E         + [('errors',
E         +   [('code', 'invalid'),
E         +    ('field', 'staff_authentication'),
E         +    ('resource', 'Login'),
E         +    ('stack_trace', None)]),
E         +  ('message', 'Staff name and password pair not match')]

tests/test_app.py:31: AssertionError
=========================== 1 failed in 0.22 seconds ===========================

How do I make the newly created JSON object to the same as the response?

Solution 1

Instead of converting the JSON response into Object, I use json.loads() to convert it into a Dictionary, and compare them.

def test_login(self, client):
        res = return client.post('/login',
            data=json.dumps(dict(staff_name='no_such_user', staff_password='password')),
            content_type='application/json',
            follow_redirects=True)
        assert res.status_code == 422
        invalid_password_json = dict(message="Staff name and password pair not match",
                                    errors=dict(
                                        resource="Login",
                                        code="invalid",
                                        field="staff_authentication",
                                        stack_trace=None,),
                                    )
        assert json.loads(res.data) == invalid_password_json

This way, I do not have to worry about whitespace differences in the JSON response, as well as ordering of the JSON structure. Simply let Python's Dictionary comparison function check for equality.

Solution 2

If you do indeed require literal, value-to-value equality between two doctionaries, it would be simpler to compare their json serialization results, otherwise you would need some recursive comparison of dicts and their values

Note: since dicts in python are unsorted collections, you would require passing sort_keys=True to json.dumps, see this question for more details

Comments

[Hanxue]

Thanks for the answer. Is sort_keys=True necessary? Even when I jumble up the keys, the comparison is still correct.

[E P]

If you use python3, it is likely due to the fact that current realization of dict is order-preseving, but as far as I know it is not intended and is not included in the specification, so it can work, but I would not rely on this in the long run. In python2 I believe this should not be the same in general, but for short dicts of fixed key list - it may work without sort_keys

[Marc]

Python3 has committed that dicts will be ordered going forward, so you can rely on it.