What's the best way to parse a JSON response from the requests library?

Question

I'm using the python requests module to send a RESTful GET to a server, for which I get a response in JSON. The JSON response is basically just a list of lists.

What's the best way to coerce the response to a native Python object so I can either iterate or print it out using pprint?

Solution 1

You can use json.loads:

import json
import requests

response = requests.get(...)
json_data = json.loads(response.text)

This converts a given string into a dictionary which allows you to access your JSON data easily within your code.

Or you can use @Martijn's helpful suggestion, and the higher voted answer, response.json().

Solution 2

Since you're using requests, you should use the response's json method.

import requests

response = requests.get(...)
data = response.json()

It autodetects which decoder to use.

Solution 3

You can use the json response as dictionary directly:

import requests

res = requests.get('https://reqres.in/api/users?page=2')
print(f'Total users: {res.json().get("total")}')

or you can hold the json content as dictionary:

json_res = res.json()

and from this json_res dictionary variable, you can extract any value of your choice

json_res.get('total')
json_res["total"]

Attentions Because this is a dictionary, you should keep your eye on the key spelling and the case, i.e. 'total' is not the same as 'Total'

Comments

[felix001]

ok great, however each of the elements would still be unicode.

[Simeon Visser]

@felix001: yes, although you can convert any data using str(). On the other hand unicode data isn't bad to have around (in preparation for the future).

[timurb]

Just keep in mind that it have appeared somewhere in between v0.12 and v1.0 so that for example Ubuntu 12.04 deb-package for python-requests does not have this function yet (it is v0.8). You can pip install requests though instead of using deb package.

[Krishna Oza]

I am a little curious what is now data is it a list of list or a dictionary.

[pswaminathan]

@Krishna_Oza data mirrors the structure of the JSON it's reading. For example, if the response is: [{"a": {...}}], data is a list, with list[0] == {'a': {...}}. If the response is {"a": "b", "c": "d"}, data is a dict. Does that answer your question?

[Martijn Pieters]

Much, much better to use response.json(), as it'll do a better job of figuring out the encoding used. (Disclaimer, I wrote some of that code).

[beep_check]

this approach is essentially the same as the json.loads(response) approach in other answers, with the exception that the latter may be easier to tab through if you're using a Jupyter environment

[CGFoX]

shouldn't it be data = response.json? Otherwise, I get TypeError: 'dict' object is not callable

[pswaminathan]

@CGFoX what version are you running? I'm still seeing the API work the same way on the latest version: >>> import requests >>> r = requests.get('http://httpbin.org/get') >>> r.json <bound method Response.json of <Response [200]>> >>> r.json() {'args': {}, ...}

[CGFoX]

I'm using Python 3.7, but I don't think that's the issue. See this thread with the same problem

[pswaminathan]

@CGFoX that is the request object, not the response. The response JSON needs to be parsed, which is why it is a method.

[Paul R.]

Upvote for this because I was usually using the json.loads(response.text) method until on some large jsons I found that using respons.json() was much faster than the other way.

[neurino]

@MartijnPieters: then how can I use requests json parser later on a memcached text of the response? i.e. having the output of response.text()?

[neurino]

@MartijnPieters, found: requests.compat.json.loads(resp_text)

[Martijn Pieters]

@neurino you want the standard library json module. The requests.compat module is there to bridge different Python versions and requests.compat.json is the same thing as json on practically every system you’d care about.

[Helen Neely]

@pswaminathan what a fantastic answer! This saved me a lot of headache. Was using the response.text which was messing this up. Thanks again :)

[sFishman]

Even with this method, I get an 'invalid control character' JSONDecodeError. Any insight into this, @Martijn Pieters?

[Martijn Pieters]

@sFishman: this method nor any other can't protect you from invalid JSON data. I can't tell you how to fix your problem because there could be any number of reasons you are fetching data that can't be decoded as JSON.

[raeldor]

Is there a better library to use, because response.json() seems to result in a data structure about 10x the size of the JSON. Why is it so big? Is it the overhead of all the contained objects?

[Alex Titov]

it is worked for my case res = requests.get('http://127.0.0.1:5000/get_classes') json_data = json.loads(res.text) print(json_data)