How can I see the entire HTTP request that's being sent by my Python application?
Solution 1
A simple method: enable logging in recent versions of Requests (1.x and higher.)
Requests uses the http.client
and logging
module configuration to control logging verbosity, as described here.
Demonstration
Code excerpted from the linked documentation:
import requests
import logging
# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
import http.client as http_client
except ImportError:
# Python 2
import httplib as http_client
http_client.HTTPConnection.debuglevel = 1
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
requests.get('https://httpbin.org/headers')
Example Output
$ python requests-logging.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org
send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Date: Sat, 29 Jun 2013 11:19:34 GMT
header: Server: gunicorn/0.17.4
header: Content-Length: 226
header: Connection: keep-alive
DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226
Solution 2
r = requests.get('https://api.github.com', auth=('user', 'pass'))
r
is a response. It has a request attribute which has the information you need.
r.request.allow_redirects r.request.headers r.request.register_hook
r.request.auth r.request.hooks r.request.response
r.request.cert r.request.method r.request.send
r.request.config r.request.params r.request.sent
r.request.cookies r.request.path_url r.request.session
r.request.data r.request.prefetch r.request.timeout
r.request.deregister_hook r.request.proxies r.request.url
r.request.files r.request.redirect r.request.verify
r.request.headers
gives the headers:
{'Accept': '*/*',
'Accept-Encoding': 'identity, deflate, compress, gzip',
'Authorization': u'Basic dXNlcjpwYXNz',
'User-Agent': 'python-requests/0.12.1'}
Then r.request.data
has the body as a mapping. You can convert this with urllib.urlencode
if they prefer:
import urllib
b = r.request.data
encoded_body = urllib.urlencode(b)
depending on the type of the response the .data
-attribute may be missing and a .body
-attribute be there instead.
Solution 3
You can use HTTP Toolkit to do exactly this.
It's especially useful if you need to do this quickly, with no code changes: you can open a terminal from HTTP Toolkit, run any Python code from there as normal, and you'll be able to see the full content of every HTTP/HTTPS request immediately.
There's a free version that can do everything you need, and it's 100% open source.
I'm the creator of HTTP Toolkit; I actually built it myself to solve the exact same problem for me a while back! I too was trying to debug a payment integration, but their SDK didn't work, I couldn't tell why, and I needed to know what was actually going on to properly fix it. It's very frustrating, but being able to see the raw traffic really helps.
Solution 4
If you're using Python 2.x, try installing a urllib2 opener. That should print out your headers, although you may have to combine that with other openers you're using to hit the HTTPS.
import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1)))
urllib2.urlopen(url)
Solution 5
A much simpler way to debug HTTP local requests is to use netcat. If you run
nc -l 1234
you'll start listening on port 1234
for HTTP connections. You can access it via http://localhost:1234/foo/foo/...
.
On the terminal, you'll see whatever raw data you sent to the endpoint. For example:
POST /foo/foo HTTP/1.1
Accept: application/json
Connection: keep-alive
Host: example.com
Accept-Language: en-en
Authorization: Bearer ay...
Content-Length: 15
Content-Type: application/json
{"test": false}
Chris B.
I write code. I run statistics. I sleep, and sometimes, I dream.
Updated on August 03, 2022Comments
-
Chris B. almost 2 years
In my case, I'm using the
requests
library to call PayPal's API over HTTPS. Unfortunately, I'm getting an error from PayPal, and PayPal support cannot figure out what the error is or what's causing it. They want me to "Please provide the entire request, headers included".How can I do that?
-
Chris B. about 12 yearsWhich of these gives me "the entire request, headers included"?
-
Skylar Saveland about 12 yearsadded some more. What else do you need besides the headers and the body?
-
Chris B. about 12 yearsI'm not entirely sure what they're looking for. I was hoping to capture everything that went over the wire for them, in that exact format, byte-for-byte.
-
BastiBen about 11 yearsThere is? Can't really find it.
-
Bruno about 11 years@badcat. There was a "Verbose Logging" section at the time. It seems it was removed in December.
-
BastiBen about 11 yearsAh, that would explain that. :) Still, now this question is kind of valid again, because I couldn't find a way to print the whole traffic between server and client for debugging.
-
cbare about 11 yearsIs there a recommended "new way" to achieve the same effect as verbose logging?
-
RayLuo almost 11 yearsSame feeling as Chris. And I end up using @Inactivist 's answer. It is good. Upvoted.
-
Inactivist almost 11 yearsMy answer demonstrates the correct method for Requests 1.x and higher.
-
Inactivist almost 11 yearsThanks, @EmmettJ.Butler =) Though I'm not sure this info was available at the time of the original inquiry.
-
Mechanical snail over 10 yearsI think this doesn't work if the initial response is a redirect.
-
Jason R. Coombs about 10 yearsNote that httplib isn't available on Python 3. To make the code portable, replace
import httplib
withimport requests.packages.urllib3.connectionpool as httplib
or use six andfrom six.moves import http_client as httplib
. -
Steve Bennett over 9 yearsI think for most cases, this is the right way to go. If you just want to see the headers, print the headers. But for the OP's particular needs, debug tracing might be justified.
-
Antti Haapala -- Слава Україні almost 9 yearsThis is the preferred way of doing it in my case. Only one note: the
response.request
seems to be aPreparedRequest
in my case; it doesn't have.data
but.body
instead. -
Flimm almost 7 yearsIn
requests
2.18.1 and Python 3, the loggerlogging.getLogger("requests.packages.urllib3")
doesn't exist or has no effect. -
Martin Tapp over 6 years
r.request.path_url
worked best for me (full_url
no longer seems to exist). -
shershen about 6 yearsfor Python3 see here - docs.python-requests.org/en/latest/api/?highlight=debug
from http.client import HTTPConnection
-
Chuck van der Linden almost 6 yearsfor the full URL (with the querystring parameters) you can also use
response.url
(which is a bit different in that it's notresponse.request...
-
user305883 about 5 yearsHow come I can't see all the request attribute you shown @Skylar ?
vars(r.request) {'_body_position': None, '_cookies': <RequestsCookieJar[]>, 'body': None, 'headers': {'User-Agent': 'python-requests/2.18.4', 'Connection': 'keep-alive', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate'}, 'hooks': {'response': []}, 'method': 'GET', 'url': 'myUrl..'}
requests==2.18.4
I am looking if actually sent request via specified proxy -
lesnik over 4 yearsUnfortunately "send:" "reply:" and "header:" lines are not actually logged, but just printed to stdout. But I want to have this info in the log files!
-
Danek Duvall almost 4 yearsThe link posted by @shershen no longer works. This appears to be the current replacement: requests.readthedocs.io/en/master/api/…
-
drkthng almost 3 yearsamazing work buddy! this just helped me a lot after hours of testing and unsuccessfully fiddling with fiddler...
-
Admin over 2 yearsto have complete logs, use 'hooks', see my answer further below
-
Shravya Boggarapu almost 2 yearsThank you!! You saved me. I needed this to replicate the functionality in a different environment. Nothing worked!! I was soo frustrated I even went through the source code of the requests library but I got lost in all the function calls.