Python Requests requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol

Question

I'm on Ubuntu 12.10 with OpenSSL 1.0.1c, python 2.7.3, Requests 1.0.3 and 1.0.4 (tried both), and when attempting to connect to the website in the url variable with the following code.

def SendInitialRequest(xmlmessage, redirecturl):
    url = 'https://centineltest.cardinalcommerce.com/maps/txns.asp'

    payload = 'cmpi_msg=' + ET.tostring(xmlmessage)
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
    }
    r = requests.post(url, data=payload, headers=headers, verify=None)
    print r.text

It throws the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "clams/libs/centinel/thinclient.py", line 134, in SendInitialRequest
    r = requests.post(url, data=payload, headers=headers, verify=None)
  File "/home/jasonamyers/.virtualenv/clams/lib/python2.7/site-packages/requests/api.py", line 87, in post
    return request('post', url, data=data, **kwargs)
  File "/home/jasonamyers/.virtualenv/clams/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jasonamyers/.virtualenv/clams/lib/python2.7/site-packages/requests/sessions.py", line 269, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/home/jasonamyers/.virtualenv/clams/lib/python2.7/site-packages/requests/sessions.py", line 364, in send
    r = adapter.send(request, **kwargs)
  File "/home/jasonamyers/.virtualenv/clams/lib/python2.7/site-packages/requests/adapters.py", line 163, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol

Attempting the connection with openssl returns the following:

$ openssl s_client -connect centineltest.cardinalcommerce.com:443
CONNECTED(00000003)
140019346777760:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 226 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

If I force it to use tls1 it works (output truncated):

$ openssl s_client -tls1 -connect centineltest.cardinalcommerce.com:443
CONNECTED(00000003)
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU
verify error:num=20:unable to get local issuer certificate
verify return:0
---

I've seen numerous bug reports for this; however, I've not found a way to get around it using the python requests library. Any assistance would be greatly appreciated.

Solution 1

Reposting this here for others from the requests issue page:

Requests' does not support doing this before version 1. Subsequent to version 1, you are expected to subclass the HTTPAdapter, like so:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl

class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1)

When you've done that, you can do this:

import requests
s = requests.Session()
s.mount('https://', MyAdapter())

Any request through that session object will then use TLSv1.

Solution 2

Setting verify=False only skips verifying the server certificate, but will not help to resolve SSL protocol errors.

This issue is likely due to SSLv2 being disabled on the web server, but Python 2.x tries to establish a connection with PROTOCOL_SSLv23 by default. This happens at https://github.com/python/cpython/blob/360aa60b2a36f5f6e9e20325efd8d472f7559b1e/Lib/ssl.py#L1057

You can monkey-patch ssl.wrap_socket() in the ssl module by overriding the ssl_version keyword parameter. The following code can be used as-is. Put this at the start of your program before making any requests.

import ssl
from functools import wraps
def sslwrap(func):
    @wraps(func)
    def bar(*args, **kw):
        kw['ssl_version'] = ssl.PROTOCOL_TLSv1
        return func(*args, **kw)
    return bar

ssl.wrap_socket = sslwrap(ssl.wrap_socket)

Solution 3

Installing the "security" package extras for requests solved for me:

sudo apt-get install libffi-dev
sudo pip install -U requests[security]

Solution 4

This is a known bug, you can work it around with a hack:

Open up site-packages/requests/packages/urllib3/connectionpool.py (or otherwise just make a local copy of requests inside your own project), and change the block that says:

def connect(self):
    # Add certificate verification
    sock = socket.create_connection((self.host, self.port), self.timeout)

    # Wrap socket using verification with the root certs in
    # trusted_root_certs
    self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
                                cert_reqs=self.cert_reqs,
                                ca_certs=self.ca_certs,
                                server_hostname=self.host,
                                ssl_version=self.ssl_version)

to:

def connect(self):
    # Add certificate verification
    sock = socket.create_connection((self.host, self.port), self.timeout)

    # Wrap socket using verification with the root certs in
    # trusted_root_certs
    self.sock = ssl_wrap_socket(sock, self.key_file, self.cert_file,
                                cert_reqs=self.cert_reqs,
                                ca_certs=self.ca_certs,
                                server_hostname=self.host,
                                ssl_version=ssl.PROTOCOL_TLSv1)

Otherwise, I suppose there's an override somewhere which is less hacky, but I couldn't find one with a few glances.

NOTE: On a sidenote, requests from PIP (1.0.4) on a MacOS just works with the URL you provided.

Solution 5

I encountered this error, and the fix appears to be turning off SNI, which Python 2.7 does not support:

http://bugs.python.org/issue5639

urllib3 on python 2.7 SNI error on Google App Engine

View more solutions

Comments

[jasonamyers]

I also had it work perfectly on MacOS, just worried about production, and that when Apple updates OpenSSL that might not always be the case. Thanks so much for the response. Gonna see if I'm actually allow to get a custom package like that pushed up to heroku :)

[favoretti]

@jasonamyers: well, you can just make it a part of your project, just copy requests folder to your own project and import requests will prefer your patched version to the stock one.

[jasonamyers]

really trying to avoid doing that for one connection out of many others :( I'll keep digging, thanks for the response.

[favoretti]

I had another deeper look, there's no way ssl_version can be overriden from the top of requests module, without modifications to the module itself :(

[jasonamyers]

This isn't an issue on Mac because they use OpenSSL 0.9.8r 8 Feb 2011, which hasn't implemented these changes yet. My guess is it's just a matter of time :(

[favoretti]

You could also downgrade SSL ;)

[jasonamyers]

It is worth noting that this same behavior occurs on a mac if you use openssl 1.0.1c :(

[Con Antonakos]

This worked for me, but now I'm getting another error: requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

[Lukasa]

For the sake of keeping this answer up to date, I did end up writing this up here. The subclass API has changed a little bit since this answer was posted, so you should use the code in that blog post, not the code in this answer. =)

[jww]

Its a bad idea to set verify=False. You need to figure out what's really broken and fix it.

[KomodoDave]

Thank you so much for providing this monkey patch - had been tearing my hair out trying to find a way to set ssl_version while using Requests library :)

[Diego Iturriaga]

Works like a charm!! But what is the scope of this change? Only my program is set to use TLS_1 or is it global?

[chnrxn]

As long as the code has been run in any Python invocation, the change will be effective. If you run another Python script/program that doesn't invoke this code, then there is no change. You could put the code into it's own module (.py file) and import this file in any script that needs this override.

[chnrxn]

@MA1, please provide more details on the problem you are facing.

[MA1]

@chnrxn I tried this but still getting issue. The weird thing is i get the issue sometime not always. I have a python3 script that makes lots of connections(not in parallel) to a site. So the workaround i have is retry to connect if SSLError exception comes and this is working.

[chnrxn]

I suspect then the root cause of your problem may be with something else. The problem in this thread is about SSL version mismatch between server and client.

[AfromanJ]

@chnrxn I have applied this patch but now get the error: urlopen error [Errno 1] _ssl.c:504: error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version what could this mean?

[chnrxn]

@AfromanJ what's the server URL? Looks like a protocol version mismatch between server and client. Can you share your relevant code as well?

[AfromanJ]

@chnrxn I have an app that scrapes council websites using python and Mechazines. I get this error on a few URLs I try and scrape, one being: planning.middevon.gov.uk/online-applications. Mechazine trys to open the url using urllib urlopen(url). Sorry if thats not enough, any help would be great, thanks.

[chnrxn]

@AfromanJ, I got a certificate verification error, which means that the protocol negotiation succeeded. It's probably because our environments are different. Having said that, this does not seem like the right place to troubleshoot this.

[Chris Johnson]

The code you referenced is is connection.py not connectionpool.py, at least in my version of requests (2.5.3).

[favoretti]

@ChrisJohnson might have changed over time :) it's been 3 years :)

[Fred Thomsen]

@chnrx python does not try to use SSLv2 exclusively in this case, it just tries to send an SSLv2-compatible ClientHello message, which may end up using SSLv2, but could use higher version as well. A server can have SSLv2 support disabled while still allowing SSLv2-compatible ClientHello messages to start the SSL negotation. But what is happening more and more is that servers are now disabling the acceptance of SSLv2-compatible ClientHello messages as well.

[Hani Goc]

I don't know why it WORKED and how it WORKED it just WORKED

[Paul]

I used this fix and now I get ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:590)

[Josh Kupershmidt]

@Lukasa's link appears to be offline, here is an archive'ed version

[sscarduzio]

This fixed a shitty SSL protocol error problem in OSX with stripe SDK, I don't know how I arrived here, but I'm adding some extra google keywords for the future me. SSLError: EOF occurred in violation of protocol (_ssl.c:590))

[socrates]

The method fixed my issue, on my Ubuntu(WSL) with Windows 10.

[Heath Raftery]

On a Beaglebone I had to first sudo apt-get install python3-dev libssl-dev for the requests[security] install to finish. It still hung for a good couple of minutes, but eventually succeeded. Alas, it made no difference to the requests SSL error.

[xjzhou]

just "pip install -U requests[security]" fixed my issue.