How do I get the IP address from a http request using the requests library?

Question

I am making HTTP requests using the requests library in python, but I need the IP address from the server that responded to the HTTP request and I'm trying to avoid making two calls (and possibly having a different IP address from the one that responded to the request).

Is that possible? Does any python HTTP library allow me to do that?

PS: I also need to make HTTPS requests and use an authenticated proxy.

Update 1:

Example:

import requests

proxies = {
  "http": "http://user:[email protected]:3128",
  "https": "http://user:[email protected]:1080",
}

response = requests.get("http://example.org", proxies=proxies)
response.ip # This doesn't exist, this is just an what I would like to do

Then, I would like to know to which IP address requests are connected from a method or property in the response. In other libraries, I was able to do that by finding the sock object and using the getpeername() function.

Solution 1

It turns out that it's rather involved.

Here's a monkey-patch while using requests version 1.2.3:

Wrapping the _make_request method on HTTPConnectionPool to store the response from socket.getpeername() on the HTTPResponse instance.

For me on python 2.7.3, this instance was available on response.raw._original_response.

from requests.packages.urllib3.connectionpool import HTTPConnectionPool

def _make_request(self,conn,method,url,**kwargs):
    response = self._old_make_request(conn,method,url,**kwargs)
    sock = getattr(conn,'sock',False)
    if sock:
        setattr(response,'peer',sock.getpeername())
    else:
        setattr(response,'peer',None)
    return response

HTTPConnectionPool._old_make_request = HTTPConnectionPool._make_request
HTTPConnectionPool._make_request = _make_request

import requests

r = requests.get('http://www.google.com')
print r.raw._original_response.peer

Yields:

('2a00:1450:4009:809::1017', 80, 0, 0)

---

Ah, if there's a proxy involved or the response is chunked, the HTTPConnectionPool._make_request isn't called.

So here's a new version patching httplib.getresponse instead:

import httplib

def getresponse(self,*args,**kwargs):
    response = self._old_getresponse(*args,**kwargs)
    if self.sock:
        response.peer = self.sock.getpeername()
    else:
        response.peer = None
    return response


httplib.HTTPConnection._old_getresponse = httplib.HTTPConnection.getresponse
httplib.HTTPConnection.getresponse = getresponse

import requests

def check_peer(resp):
    orig_resp = resp.raw._original_response
    if hasattr(orig_resp,'peer'):
        return getattr(orig_resp,'peer')

Running:

>>> r1 = requests.get('http://www.google.com')
>>> check_peer(r1)
('2a00:1450:4009:808::101f', 80, 0, 0)
>>> r2 = requests.get('https://www.google.com')
>>> check_peer(r2)
('2a00:1450:4009:808::101f', 443, 0, 0)
>>> r3 = requests.get('http://wheezyweb.readthedocs.org/en/latest/tutorial.html#what-you-ll-build')
>>> check_peer(r3)
('162.209.99.68', 80)

Also checked running with proxies set; proxy address is returned.

---

Update 2016/01/19

est offers an alternative that doesn't need the monkey-patch:

rsp = requests.get('http://google.com', stream=True)
# grab the IP while you can, before you consume the body!!!!!!!!
print rsp.raw._fp.fp._sock.getpeername()
# consume the body, which calls the read(), after that fileno is no longer available.
print rsp.content  

---

Update 2016/05/19

From the comments, copying here for visibility, Richard Kenneth Niescior offers the following that is confirmed working with requests 2.10.0 and Python 3.

rsp=requests.get(..., stream=True)
rsp.raw._connection.sock.getpeername()

---

Update 2019/02/22

Python3 with requests version 2.19.1.

resp=requests.get(..., stream=True)
resp.raw._connection.sock.socket.getsockname()

---

Update 2020/01/31

Python3.8 with requests 2.22.0

resp = requests.get('https://www.google.com', stream=True)
resp.raw._connection.sock.getsockname()

Solution 2

Try:

import requests

proxies = {
  "http": "http://user:[email protected]:3128",
  "https": "http://user:[email protected]:1080",
}

response = requests.get('http://jsonip.com', proxies=proxies)
ip = response.json()['ip']
print('Your public IP is:', ip)

Comments

[gawry]

It's not pretty but it works. Thanks! That helped a lot!

[MattH]

There's a "fine" tradition of monkey patching... The requests library is a high level abstraction of HTTP, it doesn't consider the IP address that served a response to be needful at its abstraction level. The minimal monkey patch was an interesting challenge. If you want pretty you'd have to use a lower level HTTP library, but do more HTTP logic yourself.

[gawry]

I've just realized that it doesn't work in all cases. Is not working for wheezyweb.readthedocs.org/en/latest/…. Can you tell a reason?

[gawry]

It didn't work for me. But I've managed to find the sock by changing your code to, response.fp._sock: response.peer = response.fp._sock.getpeername()

[MattH]

Perhaps a difference between versions.

[Mark]

Great solution: thanks. I had a problem with HTTPS connections: the "WrappedSocket" object didn't have the getpeername() method. I eventually found the method on self.sock.socket, so a two-step try/except for self.sock.getpeername() then self.sock.socket.getpeername() does the job for me: working on my web monitor service.

[Hywel Thomas]

Thanks. This helped me show that inconsistent results from an API were not due to load balancing.

[est]

@MattH I found a new way without the need to monkeymatch: rsp=requests.get(..., stream=True);ip=rsp.raw._fp.fp._sock.getpeername();print rsp.content see github.com/kennethreitz/requests/issues/2158

[MattH]

@est Interesting! Added it into the body here. Don't know if it was possible back with 1.2.3, but certainly works with 2.5.1 that I have to hand.

[Richard Kenneth Niescior]

I've just needed this and I have discovered that rsp=requests.get(..., stream=True);rsp.raw._connection.sock.getpeername() works. I have requests 2.10.0 and am using Python 3

[Suresh Kumar]

@RichardKennethNiescior - your solution works but when requesting via a proxy the IP address returned is the proxy's ip address but not the origin server IP address. Any idea how to get the origin server's ip address?

[MattH]

@SureshKumar: If you're using a HTTP proxy then your client never sees the IP address of the service beyond the proxy, your client asks the proxy by URL. If you're using an HTTPS proxy, then your client resolves the IP address and asks the proxy for a CONNECT to the resolved address and the HTTPS port. The proxy may add a header for an HTTP request, but may be a feature / setting.

[piroux]

It seems to be resp.raw._connection.sock.socket.getsockname() now with requests version '2.19.1'. Note that in my case, resp.raw._connection.sock is of type urllib3.contrib.pyopenssl.WrappedSocket.

[no nein]

How would I go about adding a proxy IP to the solution for Python 3?

[Andrea Bisello]

on python 3.8 requests 2.22.0 call.raw._connection.sock does not have socket property even i'm using stream=True. for example getta = requests.get("google.it", stream=True) print(getta.raw._connection.sock.socket.getsockname()) returns AttributeError: 'SSLSocket' object has no attribute 'socket' i'm not able to find where socket is.

[MattH]

@AndreaBisello Updated. sock was the socket.

[λuser]

@MattH getsockname() is the local name, getpeername() is the remote name, you're mixing both in your reply with your updates.

[Brandon Stivers]

That's great and all, but how does this work when trying to obtain that local and/or remote ip when pulling a different url which 99.9% of people using requests will be doing? This works in a testing environment when you're testing basic code. But in production, you would never do this.

[0xc0de]

You are doing a great job keeping this answer updated!