curl fails to retrieve HTTPS content: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Question

I'm trying to access the website https://www.lawsociety.com.au with curl on Windows 10 and Ubuntu 16.04. It works on Ubuntu, but fails on Windows with the message error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure. I'm not sure what is wrong and how to fix it.

Here is the curl output on Windows machine:

> curl -i -v -I https://www.lawsociety.com.au
* Rebuilt URL to: https://www.lawsociety.com.au/
*   Trying 125.7.104.7...
* TCP_NODELAY set
* Connected to www.lawsociety.com.au (125.7.104.7) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: D:\dev\curl\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

curl version:

curl 7.57.0 (x86_64-pc-win32) libcurl/7.57.0 OpenSSL/1.1.0g (WinSSL) zlib/1.2.11 WinIDN libssh2/1.8.0 nghttp2/1.28.0
Release-Date: 2017-11-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy MultiSSL    

Same command on Ubuntu box:

$ curl -I -v https://www.lawsociety.com.au
* Rebuilt URL to: https://www.lawsociety.com.au/
*   Trying 125.7.104.7...
* Connected to www.lawsociety.com.au (125.7.104.7) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.0 / RSA_3DES_EDE_CBC_SHA1
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.lawsociety.com.au (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=AU,postalCode=2000,ST=NSW,L=Sydney,street=170 Phillip Street,O=THE LAW SOCIETY OF NEW SOUTH WALES,OU=PremiumSSL Wildcard,CN=*.lawsociety.com.au
*        start date: Fri, 17 Mar 2017 00:00:00 GMT
*        expire date: Mon, 16 Apr 2018 23:59:59 GMT
*        issuer: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Organization Validation Secure Server CA
*        compression: NULL
* ALPN, server did not agree to a protocol
> HEAD / HTTP/1.1
> Host: www.lawsociety.com.au
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 26 Dec 2017 09:04:02 GMT
Date: Tue, 26 Dec 2017 09:04:02 GMT
< Server: Oracle-Application-Server-11g
Server: Oracle-Application-Server-11g
< Cache-Control: no-cache
Cache-Control: no-cache
< Content-Length: 36272
Content-Length: 36272
< Set-Cookie: JSESSIONID=kGs3hCQCW7FPhQ2Lh0JvKn9JvXhhHCK2GQKXvLps308Ww1D70pMp!1826685759; path=/; HttpOnly
Set-Cookie: JSESSIONID=kGs3hCQCW7FPhQ2Lh0JvKn9JvXhhHCK2GQKXvLps308Ww1D70pMp!1826685759; path=/; HttpOnly
< X-ORACLE-DMS-ECID: 005OJeGQSZb9xWGayxQ_MG0007Z60000EU
X-ORACLE-DMS-ECID: 005OJeGQSZb9xWGayxQ_MG0007Z60000EU
< X-Powered-By: Servlet/2.5 JSP/2.1
X-Powered-By: Servlet/2.5 JSP/2.1
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8

<
* Connection #0 to host www.lawsociety.com.au left intact

I tried openssl s_client command on Windows:

> openssl s_client -connect www.lawsociety.com.au:443
CONNECTED(00000224)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=AU/postalCode=2000/ST=NSW/L=Sydney/street=170 Phillip Street/O=THE LAW SOCIETY OF NEW SOUTH WALES/OU=PremiumSSL Wildcard/CN=*.lawsociety.com.au
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
... <removed to save space> ...
-----END CERTIFICATE-----
subject=/C=AU/postalCode=2000/ST=NSW/L=Sydney/street=170 Phillip Street/O=THE LAW SOCIETY OF NEW SOUTH WALES/OU=PremiumSSL Wildcard/CN=*.lawsociety.com.au
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 5683 bytes and written 621 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 8198FE887E4FC12D68E2388D4C052ABF
    Session-ID-ctx:
    Master-Key: 93688C15A75E3E9F596AF96DFF72B557AD28A3FEF8764401CBD12D1F432EAF4D216595D74338AF24498AB29FF5ABE759
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1514278771
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

So connection seems to be fine. Opening the URL in a browser works just fine, too.

What am I missing?

UPDATE

1. One of the possible problems is that the website is using outdated RC4-SHA cipher. I tried to enable it explicitly with curl, but curl rejects it:

> curl -i -v -I --ciphers "RC4-SHA" https://www.lawsociety.com.au
* Rebuilt URL to: https://www.lawsociety.com.au/
*   Trying 125.7.104.7...
* TCP_NODELAY set
* Connected to www.lawsociety.com.au (125.7.104.7) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* failed setting cipher list: RC4-SHA
* Closing connection 0
curl: (59) failed setting cipher list: RC4-SHA

2. Another potential problem is that root certificates might not be available. However, curl comes with the latest Mozilla CA bundle (curl-ca-bundle.crt), so I believe it is using correct certificates.

I also copied all public certificates from working Ubuntu box to the Windows machine and specified cert path to curl using --capath param - it doesn't help.

3. Just for completeness sake, I tried with the latest Python 3.6.4:

import urllib.request
with urllib.request.urlopen('https://www.lawsociety.com.au/') as u:
    print(u.read())

Python SSL should use Windows facilities for HTTPS. However, it fails with the same error:

ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:777)
...
During handling of the above exception, another exception occurred:
...
urllib.error.URLError: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:777)>

So it looks like the problem is not with missing certificate, but happens somewhere earlier. I'm not an SSL expert, so I may be wrong.

SOLUTION

The root cause of the problem is outdated SSL protocol+ciphers used by the website. In order to make curl work I downgraded it to the version that uses OpenSSL 1.0.2, which still supports RC4 ciphers. After that everything works flawlessly.

For those who need Python solution:

import ssl
import urllib.request

ctx = ssl.SSLContext(protocol=ssl.PROTOCOL_SSLv3)
ctx.set_ciphers('SSLv3')
# alternatively, for this particular website:
#ctx.set_ciphers('RC4-SHA:RC4-MD5')

with urllib.request.urlopen('https://www.lawsociety.com.au/', context=ctx) as u:
    print(u.read())

Solution

I believe there are actually two somewhat overlapping problems here, both related to the server apparently being way out of date or else insanely operated. Analysis by ssllabs shows it supports only SSLv3 and TLSv1.0 protocols, and only four ciphersuites:

TLS_RSA_WITH_DES_CBC_SHA (0x9)   INSECURE   56
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa)   WEAK  112
TLS_RSA_WITH_RC4_128_MD5 (0x4)   INSECURE   128
TLS_RSA_WITH_RC4_128_SHA (0x5)   INSECURE   128

(which in the OpenSSL naming scheme are DES-CBC-SHA, DES-CBC3-SHA, RC4-MD5 and RC4-SHA).

First, as noted by SSLLabs the server is "version intolerant"; if you send it ClientHello offering versions above 1.0 in a record with version 1.0 (and otherwise acceptable) it negotiates down to 1.0 as it should*, but if you send this offer with record version 1.1 or 1.2, as some software does (but not AFAICT any recent OpenSSL), the server aborts with alert close_notify (which is not correct for this state). (*: well, as it should given that it's supporting 1.0 at all)

Second, it supports only the four ciphersuites above. Very recent versions of OpenSSL and specifically 1.1.0g no longer support single-DES (at all) because it is completely broken, and do not support RC4 or triple-DES by default because of various biases and the generic birthday attack; thus if OpenSSL is used there are no ciphersuites in common and the server correctly aborts with alert handshake_failure. But it should be possible to enable RC4 and/or TDES, unless they were configured out at build (compile) time. I notice your Windows version shows (WinSSL) in addition to OpenSSL/1.1.0g; I don't know if this means schannel is used here (or at all), in which case it might be the reason --ciphers RC4-SHA (using the OpenSSL naming scheme) didn't work.

Note on Windows programs, especially 'imports' like curl and OpenSSL, typically don't share libraries as they do on Unbuntu and most Linux (and other Unix). You might try openssl version on your Windows OpenSSL to see which it is; I'll bet it's not the latest and if you get the latest it will exhibit the same issue from commandline.

Ubuntu 16.04 (I assume LTS?) is not bleeding edge and it makes sense for it to still support ciphers that were deprecated only recently.

Comments

[Asim]

In the python solution, I had to change ctx = ssl.SSLContext(protocol=ssl.PROTOCOL_SSLv3) to ctx = ssl.SSLContext(protocol=ssl.PROTOCOL_TLS). I'm on Ubuntu

[Alex Blekhman]

thanks for the link. According to OpenSSL test client, it seems that the website is using RC4-SHA cipher. I tried to specify cipher explicitly with curl command, but it doesn't help. See my update of the original post.

[Alex Blekhman]

Thank you for the detailed analysis of the problem. My SSL knowledge is cursory at best, so your answer is very educational. You're spot on about unsupported ciphers in recent OpenSSL releases. My Windows OpenSSL version was 1.0.2n and that's why it could establish connection. Ubuntu box has version 1.0.2g and older curl 7.47.0. I tried the same version of curl for Windows and it worked flawlessly, thanks to older OpenSSL the curl executable was built with, I presume. The moment I tried the latest OpenSSL 1.1.0g on Windows, it failed with exactly same error: sslv3 alert handshake failure.