python3 and requests: still getting 'sslv3 alert handshake failure'

Question

I have been trying to perform an HTTPS request in Python 3 using requests and aggregating pretty much all the knowledge from the prior attempts documented on StackOverflow. I cannot for the life of me seem to get out of the sslv3 alert handshake failure rabbit hole.

This is my environment:

• macOS 10.13.6
• Python 3.7.0 (installed via Homebrew along with openssl)
• OpenSSL 1.0.2p 14 Aug 2018 (output of print(ssl.OPENSSL_VERSION))
• requests 2.19.1 (output of print(requests.__version__)) installed via pip install requests[security]
• even cryptography 2.3.1 is installed

This is the bare-bone failing code:

>>> import requests
>>> requests.get('https://iris.nuigalway.ie')

And this is the output:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3-1.23-py3.7.egg/urllib3/contrib/pyopenssl.py", line 444, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.7/site-packages/pyOpenSSL-18.0.0-py3.7.egg/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.7/site-packages/pyOpenSSL-18.0.0-py3.7.egg/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.7/site-packages/pyOpenSSL-18.0.0-py3.7.egg/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]

Needless to say it works with cURL, browsers etc.

curl --verbose "https://iris.nuigalway.ie"

Here's a handshake snippet of the ouput:

* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS change cipher, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / DES-CBC3-SHA
* ALPN, server did not agree to a protocol

Now the cipher used by cURL does not indeed seem to be among the default ciphers of urllib3 1.23 (seemingly used by requests) as per https://github.com/urllib3/urllib3/blob/1.23/urllib3/util/ssl_.py

So I tried adding it using the advice given at https://stackoverflow.com/a/40741362 like this:

>>> requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += 'DES-CBC3-SHA'

and even setting it to ALL. I even tried not to verify the certificate, all to no avail.

>>> requests.get('https://iris.nuigalway.ie', verify=False)

A check with s_client on the server:

$ openssl s_client -connect iris.nuigalway.ie:443

reveals the following TLS version and cipher:

New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA

What options could I possibly have not tried yet?

Many Thanks

UPDATE

The values of ssl.OPENSSL_VERSION OpenSSL.SSL.SSLeay_version(0) revealed two different versions of OpenSSL used by ssl and pyOpenSSL respectively, the latter being a more recent OpenSSL 1.1.0i 14 Aug 2018 that has most likely dropped support for the DES-CBC3-SHA cipher.

Below is the temporary solution I have adopted:

1. uninstall cryptography
2. injecting only the required cipher, like this:

(Note that it is no longer a concatenation)

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'DES-CBC3-SHA'

I appreciate that this solution may be sub-optimal and not applicable to many cases, but at least the lesson learnt is that different versions of OpenSSL may be at play from one package to another.

I will be happy to know of a more flexible solution if any.

Solution

The server seems to be really broken. If you just add DES-CBC3-SHA to the list of ciphers it will not work, maybe because the server croaks because the client offers ciphers the server does not know or because of too much ciphers.

If one instead changes it to only allow this single cipher it works for me, i.e.

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'DES-CBC3-SHA'

Comments

[Patrick Mevzek]

You shouldn't have to mess with ciphers, and you certainly should not deactivate certificate validation. From your output it seems the server may speak only TLS 1.0 and not higher versions, so first try that. See "Example: Specific SSL Version" in docs.python-requests.org/en/master/user/advanced . Also make sure that you use the same CA bundle.

[CristiFati]

* ALPN, server did not agree to a protocol. Maybe the server's SSL implementation doesn't know ALPN? Try building OpenSSL without ALPN support (if possible), or use an older version (e.g. 1.0.1 - which is no longer supported).

[alexdma]

@PatrickMevzek thanks, I tried something along those lines that I'm not sure is right though (and has led me to the same error as before): class Tls1HttpAdapter(HTTPAdapter): def init_poolmanager(self, connections, maxsize, block=False): self.poolmanager = PoolManager( num_pools=connections, maxsize=maxsize, block=block, ssl_version=ssl.PROTOCOL_TLSv1) then I mounted a requests.Session() with that Tls1HttpAdapter() and tried a get on that session, to no avail.

[lockcmpxchg8b]

Just for paranoia's sake, can we see the output of print(requests.get('https://www.howsmyssl.com/a/check', verify=False).json()['tls_version']) just to make sure requests is trying a sane client?

[alexdma]

Apart from the insecure request warning, the output is 'TLS 1.2'

[alexdma]

Thanks, though I get another error when trying that: OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_set_cipher_list', 'no cipher match')] as if there was no such cipher in OpenSSL 1.0.2p. And yet it appears when I list them via /usr/local/opt/openssl/bin/openssl ciphers Was your environment otherwise the same as mine?

[Steffen Ullrich]

@alexdma: my environment was not the same as yours but I've used a openssl which has support for 3DES. Could it be that your /usr/local/opt/openssl/bin/openssl is not the same as the openssl library you use in Python?

[alexdma]

They are at least of the same version: /usr/local/opt/openssl/bin/openssl version and >>> ssl.OPENSSL_VERSION both print OpenSSL 1.0.2p 14 Aug 2018.

[Steffen Ullrich]

@alexdma: then I don't know - if you say that openssl version shows DES-CBC3-SHA as available you should also be able to set and use it as DEFAULT_CIPHERS as long as the same openssl library is used in both cases.

[alexdma]

Is it at all possible that the ssl module that I import when checking ssl.OPENSSL_VERSION may be different from whatever the pyOpenSSL-18.0.0 on which urllib3 and request depend is using?

[Steffen Ullrich]

@alexdma: maybe. Check OpenSSL.SSL.SSLeay_version(0)

[Roger Filmyer]

"maybe because the server croaks because the client offers ciphers the server does not know" - As of early 2019, you can easily test this because the server will not work with Chromium-based browsers or with iOS 14 or newer thanks to the addition of TLS GREASE