HTTPS proxies not working with Python's requests module

33,527

Edit June 2019: This reply is not relevant anymore. Issue(s) are fixed.

Edit 2: "note that even for https proxy, the proxy address' scheme is http, it's because the client and proxy server initiate the tunnelling(the CONNECT method) in plain http. However, that's may not be true 3 years ago." - From the comments

HTTPS is 'bugged' in requests. I don't know the specifics but you can find a few other topics on this website concerning the issue. Also a Github issue is still active here. I'm suspecting you're having the problems mentioned there. If I am totally wrong, someone correct me.

To verify:

$~ curl --proxy https://27.254.52.99:8080 icanhazip.com
27.254.52.99

Works, but then in Python:

>>> proxies={'https': 'https://27.254.52.99:8080'}
>>> r = requests.get('http://icanhazip.com', headers={'User-Agent': 'Bla'}, proxies=proxies)
print r.content
<my ipv6 address comes up>

As you can see, my address comes up which means the proxy did nothing.

I don't understand why you are receiving a stacktrace. Maybe because your API is on HTTPS as well (?). Or maybe your API is just... down.

Anyway, the proxy does work in requests if its over HTTP.

>>> proxies={'http': 'http://27.254.52.99:8080'}
>>> r = requests.head('http://icanhazip.com', headers={'User-Agent': 'Bla'}, proxies=proxies)
print r.content
27.254.52.99
Share:
33,527
Lance
Author by

Lance

Updated on July 21, 2022

Comments

  • Lance
    Lance almost 2 years

    I'm pretty new to Python and I've been using their requests module as a substitute for PHP's cURL library. My code is as follows

    import requests
    import json
    import os
    import urllib
    import math
    import sys
    
    def main() :    
       url = 'https://api.com'
    
       headers = {'Content-Type': 'application/json; charset=utf-8',
                  'User-Agent': '(iPhone; iOS 7.0.4; Scale/2.00)'}
    
       d = {'token': "12345"}
    
       proxies = {
          "https": "https://27.254.52.99:8080",
       }
    
       post = json.dumps(d);
       r = requests.post(url, data=post, headers=headers, proxies=proxies)
       print r.json
    
    if __name__ == "__main__":
        main()
    

    However, I'm greeted with the following error:

    File "test.py", line 42, in test
    r = requests.post(url, data=post, headers=headers, proxies=proxies)
    File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
    File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
    File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
    File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
    File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/adapters.py", line 381, in send
    raise ProxyError(e)
    ProxyError: Cannot connect to proxy. Socket error: [Errno 54] Connection reset by peer.