SSLHandshakeException: Received fatal alert: handshake_failure after Java 6 -> 8 upgrade

Question

We've recently updated a project from Java 6 to Java 8 and now we've hit a brick wall regarding SSL handshake.

The service layer uses a client to request and receive calls from a third party application. In the service layer, the keystore is initialized with

    System.setProperty("javax.net.ssl.trustStore", keyStoreFile);
    System.setProperty("javax.net.ssl.trustStorePassword", keyStorePassword);

and injected via applicationContext.xml:

    <property name="keyStoreFile" value="/keystore/keystore.keystore" />
    <property name="keyStorePassword" value="password" />

The client is supposed to trust all certificates in case of errors:

private void trustHttpsCertificates() throws Exception {

    try {
        Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());

        // Create a trust manager that does not validate certificate chains 
        TrustManager[] trustAllCerts = new TrustManager[] { 
            new X509TrustManager() {
                public X509Certificate[] getAcceptedIssuers() {
                    return new X509Certificate[0];
                }

                public void checkServerTrusted(X509Certificate[] certs, String authType) {
                }

                public void checkClientTrusted(X509Certificate[] certs, String authType) {
                }
            }
        };

        // Ignore differences between given hostname and certificate hostname 
        HostnameVerifier hv = new HostnameVerifier() {
          public boolean verify(String hostname, SSLSession session) { return true; }
        }; 

        // Install the all-trusting trust manager 
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCerts, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        HttpsURLConnection.setDefaultHostnameVerifier(hv);
    } catch (KeyManagementException e) {
        String errorMsg = "client initialization error: " + e.getMessage();
        log.error(errorMsg);
        throw new Exception(errorMsg, e);
    } 
}

I changed "SSL" above to "TLSv1" in the upgrade process, since SSL isn't supported on Java 8. Might that be a problem?

Debug logs

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
http-nio-9080-exec-6, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1423711122 bytes = { 237, 188, 53, 112, 79, 112, 248, 92, 164, 127, 178, 34, 205, 40, 245, 25, 77, 143, 116, 126, 203, 96, 61, 181, 114, 148, 66, 227 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
***
[write] MD5 and SHA1 hashes:  len = 237
// ...
http-nio-9080-exec-6, READ: TLSv1 Alert, length = 2
http-nio-9080-exec-6, RECV TLSv1.2 ALERT:  fatal, handshake_failure
http-nio-9080-exec-6, called closeSocket()
http-nio-9080-exec-6, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
// ...
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)

Thoughts

There are mainly two things that stand out compared to the old version with Java 6:

In the logs for the old verison, when a successful attempt is made, it clealy presents the correct certificate:

http-9080-1, READ: TLSv1 Handshake, length = 1375
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: xxx
  Signature Algorithm: MD5withRSA

  Key:  Sun RSA public key, 1024 bits
  Validity: [From: Wed May 26 14:31:31 CEST 1999,
               To: Thu May 25 14:31:31 CEST 2000]
  Issuer: xxx
  SerialNumber: [    01]
]

Algorithm: [MD5withRSA]

This doesn't happen in the new Java 8 version.

Also, TLSv1 vs TLSv1.2?

http-nio-9080-exec-6, READ: TLSv1 Alert, length = 2
http-nio-9080-exec-6, RECV TLSv1.2 ALERT:  fatal, handshake_failure

Is it saying I'm trying to connect with TLSv1.2? Or TLSv1? And it's not accepted? I don't really understand. Is there a way to find out what TLS versions are accepted by the server?

I've tried adding flags to startup:

-Dhttps.protocols=TLSv1
-Ddeployment.security.TLSv1=true
-Djavax.net.ssl.keyStore=C:\keystore\keystore.keystore
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStore=C:\keystore\keystore.keystore
-Djavax.net.ssl.trustStorePassword=password

Also adding a keystore Manager programmatically:

KeyStore ks = KeyStore.getInstance("JKS");
InputStream ksIs = new FileInputStream("/keystore/keystore.keystore");

try {
    ks.load(ksIs, password.toCharArray());
} catch (Exception e) {
    e.printStackTrace();
} finally {
    if (ksIs != null) {
        try {
            ksIs.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(ks, password.toCharArray());

and then initializing SSLContext with that:

sc.init(kmf.getKeyManagers(), trustAllCerts, new SecureRandom());

but the problem persists. Help, anyone?

Solution 1

Alright, so we've got it working now. I'll post the answer here in case someone might need it some day.

We have tried quite a few things, so I'm not exactly sure what actually needed to be done in order for it to work, but here are some of the things we changed in the process.

 -Dhttps.protocols=SSLv3,TLSv1,SSLv2Hello

Adding this flag led to the certificate being presented in the javax.net.debug logs, but we were still getting SSLHandshakeException. It seems like the only cipher the server would accept was SSL_RSA_WITH_RC4_128_MD5. This was not picked automatically by our client.

-Dhttps.cipherSuites=SSL_RSA_WITH_RC4_128_MD5

We added this flag to restrict the cipher suits for the client. Together with setting the same restriction programmatically (not sure if both are needed):

socket.setEnabledCipherSuites(new String[] {"SSL_RSA_WITH_RC4_128_MD5"});

Restricting the available cipher suites to the only one that the client could use, made the client pick that cipher suite.

We also did the following changes the the jre/lib/security/java.security file to enable SSLv3 and the SSL_RSA_WITH_RC4_128_MD5 cipher:

• remove SSLv3 from jdk.tls.disabledAlgorithms
• add SSL_RSA_WITH_RC4_128_MD5 to jdk.tls.legacyAlgorithms

This is probably not recommended for production servers, since SSLv3 is obsolete, and the cipher is very old and outdated, but in this case security is not a huge concern (internal application use).

These posts were also helpful to me:

• SSL connection failing for Java 7
• Java 7 (acting as client) SSL handshake failure with keystore and truststore that worked in Java 6
• Received fatal alert: handshake_failure through SSLHandshakeException
• Java cipher suites

Solution 2

My guess is that your server is too broken to deal properly with TLS 1.2 handshakes. Usually a server which does not understand TLS 1.2 should reply with the best version it can but your server does not.

Broken servers like this exists and browsers try to work around these by retrying with a lower TLS version. Outside of browsers these retries are not common so these clients simply fail.

While I cannot say for sure that the server is broken the certificate which expired 15 years ago and was signed with the long broken MD5 algorithm suggest that you have to do with a very old and neglected installation. So chances are high that it never occurred to the developers of the original server that something like TLS 1.2 might ever exist or that the it croaks on one of the TLS extensions used in the TLS 1.2 handshake.

Since this issue is not related to the validation of the certificate all attempts to fix the issue by fiddling in the area of validation are useless. You might have more success if you enforce the use of TLS 1.1 or TLS 1.1 instead of TLS 1.2. You might try to do this with the -Dhttps.protocols=TLSv1,TLSv1.1 or -Dhttps.protocols=TLSv1 settings.