Python requests module: urlencoding json data

Question

I'm working on an API wrapper. The spec I'm trying to build to has the following request in it:

curl -H "Content-type:application/json" -X POST -d data='{"name":"Partner13", "email":"[email protected]"}' http://localhost:5000/

This request produces the following response from a little test server I setup to see exatly what headers/params etc are sent as. This little script produces:

uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data={"name":"Partner13", "email":"[email protected]"}

So that above is the result I want my python script to create when it hits the little test script.

I'm using the python requests module, which is the most beautiful HTTP lib I have ever used. So here is my python code:

uri = "http://localhost:5000/"
headers = {'content-type': 'application/json' }
params = {}
data = {"name":"Partner13", "email":"[email protected]"}
params["data"] = json.dumps(data)
r = requests.post(uri, data=params, headers=headers)

So simple enough stuff. Set the headers, and create a dictionary for the POST parameters. That dictionary has one entry called "data" which is the JSON string of the data I want to send to the server. Then I call the post. However, the result my little test script gives back is:

uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data=%7B%22name%22%3A+%22Partner13%22%2C+%22email%22%3A+%22example%40example.com%22%7D

So essentially the json data I wanted to send under the data parameter has been urlendcoded.

Does anyone know how to fix this? I have looked through the requests documentation and cannot seem to find a way to not auto urlencode the send data.

Thanks very much, Kevin

Solution 1

When creating the object for the data keyword, simply assign a variable the result of json.dumps(data).

Also, because HTTP POST can accept both url parameters as well as data in the body of the request, and because the requests.post function has a keyword argument named "params", it might be better to use a different variable name for readability. The requests docs use the variable name "payload", so thats what I use.

data = {"name":"Partner13", "email":"[email protected]"}
payload = json.dumps(data)
r = requests.post(uri, data=payload, headers=headers)

Solution 2

Requests automatically URL encodes dictionaries passed as data here. John_GG's solution works because rather than posting a dictionary containing the JSON encoded string in the 'data' field it simply passes the JSON encoded string directly: strings are not automatically encoded. I can't say I understand the reason for this behaviour in Requests but regardless, it is what it is. There is no way to toggle this behaviour off that I can find.

Best of luck with it, Kevin.

Comments

[t-8ch]

The data from your curl example isn't valid json. It's a mix between form-encoding and json.

[John]

The "here" you're referring to is encoding url parameters, which, because they go in the url, are of course url encoded. In the requests post function signature, "params" refers to url parameters, whereas "data" refers to the POST body.

[augurar]

@John_GG Actually this is used to encode a request body as well. The call stack is something like Request.init() --> PreparedRequest.prepare() --> PreparedRequest.prepare_body() --> RequestEncodingMixin._encode_params(). This appears to be to handle requests with application/x-www-form-urlencoded content automatically.