python requests gives 'None' response, where json data is expected

Question

Firstly, I should add that you can find this request by doing the following:

1- Go to [airline site][1]
2- Type in "From" = "syd"
3- Type in "To" = "sin"
4- Make the departure date sep.3 and click one-way and search
5- On the search result page check your network get request when you click on an available seat option radio button

I'm trying to use the requests module to get the response for example from this site

And this is what I'm trying:

url = "http://www.singaporeair.com/chooseFlightJson.form?"
payload = {'selectedFlightIdDetails[0]':amount_data,'hid_flightIDs':'','hid_recommIDs':'','tripType':"O",'userPreferedCurrency':""}

response = requests.get(url, params=payload)
print response.json()

The response is supposed to be:

{"price":{"total":"595.34","currency":{"code":"AUD","label":""},"adult":{"count":1,"label":"Adult","cost":"328.00","total":"328.00"},"child":{"count":0,"label":"Child","cost":"0.00","total":"0.00"},"infant":{"count":0,"label":"Infant","cost":"0.00","total":"0.00"},"addOns":[{"label":"Airport / Government taxes ","cost":"83.24"},{"label":"Carrier Surcharges","cost":"184.10"}],"disclaimer":"Prices are shown in Canadian Dollar(CAD)","rate":"595.34 AUD \u003d 913.80 CAD","ratehint":"Estimated","unFormattedTotal":"595.34"},"messages":{"O3FF11SYD":"A few seats left","O1FF31SYD":" ","R0FF31SYD":"A few seats left","O2FF31SYD":"A few seats left","O0FF31SYD":" ","O1FF11SYD":"A few seats left","O0FF21SYD":" ","O2FF21SYD":" ","O3FF21SYD":" ","O1FF21SYD":" "},"cabinClass":{"onwardCabin":["Economy"]}} 

Solution 1

The solution is to use requests session, like so:

session = requests.Session()

Then to call all of the urls you need to simply do:

response = session.get(url)

This sets the cookies and session variables which are necessary to retrieve the data. I have seen jsessionid used in different ways, in the url, or in this case in a cookie. But there are probably other session info that are required, which is taken care of by a requests session object.

Solution 2

The response is the value None, encoded in JSON; the server returns null\r\n, which means the same as None in Python.

The content type is wrong here; it is set to text/html, but the response.json() return value is entirely correct for what the server sent:

>>> import requests
>>> url = "http://www.singaporeair.com/chooseFlightJson.form?"
>>> amount_data = 0
>>> payload = {'selectedFlightIdDetails[0]':amount_data,'hid_flightIDs':'','hid_recommIDs':'','tripType':"O",'userPreferedCurrency':""}
>>> response = requests.get(url, params=payload)
>>> response
<Response [200]>
>>> response.headers['content-type']
'text/html; charset=ISO-8859-1'
>>> response.text
'null\r\n'
>>> response.json() is None
True