Python Requests SSL issue

Question

It's been days now since I started to look for a solution for this.

I've been trying to use requests to make an https request through a proxy with no luck.

Altough this is included in a bigger project of mine, it all boils done to this:

    import requests

    prox = 'xxx.xxx.xxx.xxx:xxx' # fill in valid proxy

    r = requests.get('https://ipdb.at', proxies={'http': prox,
                                                 'https': prox})

I tried the kward verify=False but I keep getting the same error or a variation of it:

    Traceback (most recent call last):
      File "/Users/mg/Desktop/proxy_service.py", line 21, in <module>
        verify=False)
      File "/Library/Python/2.7/site-packages/requests/api.py", line 55, in get
        return request('get', url, **kwargs)
      File "/Library/Python/2.7/site-packages/requests/api.py", line 44, in request
        return session.request(method=method, url=url, **kwargs)
      File "/Library/Python/2.7/site-packages/requests/sessions.py", line 279, in request
        resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
      File "/Library/Python/2.7/site-packages/requests/sessions.py", line 374, in send
        r = adapter.send(request, **kwargs)
      File "/Library/Python/2.7/site-packages/requests/adapters.py", line 213, in send
        raise SSLError(e)
    requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
    [Finished in 20.4s with exit code 1]

I'm using the latest versions of requests and openssl and 2.7.3 for python. I tested it on mac mountain lion 10.8.2 and also on windows seven.

I googled this out, saw others having similar issues. I found related pull requests on the libraries requests and urllib3, also saw information about this being a bad implementation of the http connect verb. I found a fix to use a custom adapter (i don't recall exactly but I think it's the right term) to use a custom ssl protocol. Tried them all, none helped.

So, I'm looking for more info. Do you have any idea what is going on, I can I fix it, etc, etc...

All help is welcome and thank you all!

PS: I tried several proxies, and I'm sure they are not the issue.

Solution 1

I had this same problem but when trying to use oauth2client (Google library for using oauth2 against their APIs). After a day of faffing ("experimentation"), I discovered that I had the env variable set as follows:

https_proxy=https://your-proxy:port

Changing this to

https_proxy=http://your-proxy:port

completely fixed the problem for me. Presumably this means that the initial handshake from the client to the proxy is now unencrypted, but after that the https must flow directly. I don't believe my company proxy was configured to talk https anyway.

Note however that my browsers all behaved fine with the previous setting. Perhaps they silently tried to establish the proxy connection over http after failing to use https.

Googling shows examples of setting https_proxy to both http or https addresses in equal measure. I guess it depends on how the proxy is configured.

The "use for all protocols" button in browser/OS configuration pop-ups makes it easy to accidentally use the setting I had.

However I'm not sure if that is your problem. Your code suggests that you already use exactly the same proxy setting for both http and https.

By the way, this was all on Ubuntu 12.04.1.

Solution 2

The answer here worked for me using requests with TLS doesn't give SNI support

I Installed the three dependent libraries and included the monkey patch tip in the second answer

Comments

[Shadow The Kid Wizard]

Please stop posting and deleting. You should be able to see your deleted post so edit it then undelete.

[Roman Rdgz]

This has worked for me. I don't know if this is extensible to any kind of proxy for https protocol out there, but if so, requests library should be able to detect and correct this issue by itself