Python Requests POST doing a GET?

Question

I am using Python 2.7.5, Django 1.7, requests 2.4.1, and doing some simple testing. However, it seems like when I call requests.post, the method is doing a GET instead.

My code, talking to a RESTful API. Note that the POST command works via Hurl.it with this payload and endpoint:

def add_dummy_objective(self):
    """
    To the bank
    """
    payload = {
        'displayName': {
            'text': self._test_objective
        },
        'description': {
            'text': 'For testing of API Middleman'
        },
        'genusTypeId': 'DEFAULT'
    }
    obj_url = self.host + self.bank_id + '/objectives/?proxyname=' + self._admin_key
    req = requests.post(obj_url, data=json.dumps(payload), headers=self.headers)
    return req.json()

I am setting the headers to json:

self.headers = {
    'Content-Type'  : 'application/json'
}

Instead of creating a new objective (as expected with a POST), I get a list of objectives back (what I would expect with a GET). Using pdb, I see:

(Pdb) req.request
<PreparedRequest [GET]>
(Pdb) req.request.method
'GET'

How did this get flipped? I have used the Python requests library before with no issues, so I'm not sure if I am missing something obvious or if (with newer versions of Django / Requests) I have to set another parameter? Is this a caching issue? Any tips for debugging? I have tried re-installing requests, and rolling back Django to 1.6.5, but nothing works...must be simple. -- Thanks!

====== UPDATE 1 ========

Just consolidating some of the debug info that Martijn suggested here:

(Pdb) requests.post.__name__
'post'

Stepping into the requests/api.py > post() definition:

(Pdb) l
 88         :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
 89         :param \*\*kwargs: Optional arguments that ``request`` takes.
 90         """
 91         import pdb
 92         pdb.set_trace()
 93  ->     return request('post', url, data=data, **kwargs)

Drilling down into the request() method:

(Pdb) method
'post'
(Pdb) l
 43           >>> req = requests.request('GET', 'http://httpbin.org/get')
 44           <Response [200]>
 45         """
 46         import pdb
 47         pdb.set_trace()
 48  ->     session = sessions.Session()
 49         return session.request(method=method, url=url, **kwargs)

One more layer, in session.request:

(424)request()
-> method = builtin_str(method)
(Pdb) method
'post'
(Pdb) l
419             :param cert: (optional) if String, path to ssl client cert file (.pem).
420                 If Tuple, ('cert', 'key') pair.
421             """
422             import pdb
423             pdb.set_trace()
424  ->         method = builtin_str(method)
425
426             # Create the Request.
427             req = Request(
428                 method = method.upper(),
429                 url = url,

Stepping down to the end of the method, where the request is actually made, my "prep" is a POST, but my resp is a GET:

(Pdb) prep
<PreparedRequest [POST]>
(Pdb) n
-> return resp
(Pdb) resp
<Response [200]>
(Pdb) resp.request
<PreparedRequest [GET]>
(Pdb) l
449                 'allow_redirects': allow_redirects,
450             }
451             send_kwargs.update(settings)
452             resp = self.send(prep, **send_kwargs)
453
454  ->         return resp
455
456         def get(self, url, **kwargs):
457             """Sends a GET request. Returns :class:`Response` object.
458
459             :param url: URL for the new :class:`Request` object.

Solution 1

To be clear, whenever requests receives a redirect (with a certain status code) we have to perform certain transformations on the request.

In cases like this, when you see something very unexpected the best debugging tips are to retry your request but with allow_redirects=False. This will immediately return the 30x response. Alternatively, you can also check r.history to see if there are any 30x responses that were followed. In this case you probably would have seen something like

>>> r.request.method
'GET'
>>> r.history
[<Response [302]>,]
>>> r.history[0].request.method
'POST'

We know that doing this can cause unexpected behaviour for users (as it just did to you) but it's the only correct way to operate on the web.

I hope this helps you understand why this happened beyond the fact that it was a redirect, and hopefully it gives you and others tools to debug this in the future.

Solution 2

Thanks to Martijn for some debugging tips! The problem was the RESTful API was redirecting me from http:// to https://, which caused the library to return the "second" request (GET)...

Comments

[Martijn Pieters]

The code that you posted will not produce a GET, it'll produce a POST. Are you 100% certain you are executing the right code path?

[user]

This is part of my unit testing, and I am stepping through it line by line, so I am 99% certain this is the code path it follows -- if there is some other way to check to eliminate that 1%, I am ignorant of how but would be happy to do it...

[user]

By the way, I agree and understand that the above code should produce a POST...so I assume something else is wrong (the 1%). What other ways can I find the path I am executing (besides pdb), if I am using the wrong code path?

[Martijn Pieters]

Perhaps some code elsewhere did requests.post = requests.get. What does requests.post.__name__ produce?

[Martijn Pieters]

Step into the requests.post call, check the surrounding context with l, step in further and see what method is used for the request being sent, etc.

[user]

Thanks Martijn, I will try some of those things and post the results in the question. requests.post.__name__ returns 'post'...

[user]

Thanks for the clear explanation! I did some reading up on redirects after finding this, and better understand why the library behaves as it does...

[Ethan Brouwer]

For any future readers, the allow_redirects=False check and then printing out everything works really well at diagnosing what could be causing the redirect to happen, because if you check the information after the redirect happens, it will be different.

[Liza]

This was exactly my issue