SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error on mutual authendication

Question

I have configured JBOSS 5 with SSLVerifyClient="require"

1. Client contains, server CA certs and there own certs(JDK 1.8).
2. Server contains, client CA certs and there own certs(JDK 1.6).

For this case both CA's are different, when we are trying to communicate to that server and getting tlsv1 alert decrypt error

I have no clues what was the exception it is ?

During Curl,

Unknown SSL protocol error in connection

During OpenSSL:

SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A 
SSL_connect:error in SSLv3 flush data 
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

Solution

The issue is probably (estimate based on OpenSSL output) that used algorithm mode is not supported by other side, or is considered deprecated/breached (i.e. TLSv1 is considered deprecated for more than a year).

I would recommend to capture handshake with tcpdump/wireshark. Since it seems that handshake is failing, you should be able to look into unencrypted handshake phase. Otherwise, since you have access to the private keys for the certs, you should be able to decrypt captured communication in wireshark too.

If supported-algorithm mismatch is not the issue, my second guess is DNS related. In some cases, it is not enough that client proved his identity with certificate, but there has to be a match among for example his DNS record (PTR) and CommonName field in the certificate.

Comments

[Shankar]

Is java 6 supports SHA256RSA ?

[dave_thompson_085]

(1) Wow that's old (2) Yes j6 supports SHA256withRSA -- and even if it didn't it couldn't cause this alert (3) which end is reporting the alert? (4) Is your server in fact using Java SSL (JSSE), or 'native' = ApacheAPR = OpenSSL? Jboss supports both, although the method(s) to select it varied with version and I don't remember specifics for 5. (5) Are you sure there isn't anything in the network 'between' your client and server that might be modifying the data, such as an IDS, IPS, DLP, WAF?

[Shankar]

Yes j6 supports SHA256withRSA - Do we have any specific build version ? This error while communication Mutual Authentication, but it is not occurring one way authentication.

[dave_thompson_085]

It doesn't depend on version, and in any case as I said any problem involving an RSA signature could not cause this alert (although it could cause others). It makes no sense to me for this to be related to mutual auth, unless there's some hidden internal linkage like a mismatched callback corrupting something. I would still check the network first, then compare traces or if possible logs from both ends.

[dave_thompson_085]

That alert is not caused by 'mode' (?) or protocol unsupported, which have different alerts, or anything about the cert, which also do. Wireshark is a possibility, or if using Java (see my comment on Q) set sysprop javax.net.debug=ssl (which decodes for you, and includes some endpoint state and events as well as the wire data). Especially if OP can test Java client (even a dummy test client) to Java server and compare (JSSE) key computation at both ends. ...

[dave_thompson_085]

... PS: that openssl s_client output does NOT mean TLSv1(.0) was used, only that the handshake produced SSL3/TLS1 format data in the 'struct ssl_st'. The actual protocol version would be in a different line that may be omitted for unsuccessful handshake. Also neither j6 nor j8, nor any OpenSSL to date, considers TLS1.0 obsolete (though some authorities like PCISSC do).

[Shankar]

Is there any options to support both SHA1 and SHA2 certificate on both JBOSS 5.1.0 ?(JDK 1.6.x) client and server ?

[dave_thompson_085]

@Shankar: Jboss doesn't know about cert details at all. If you are using JSSE on Jboss, (any) JDK1.6 supports (all) SHA2-RSA -- at least by default; I think Oracle backported the disableAlgorithm stuff to later, paid-support versions of 6, and IF so and IF that is configured wrong it could cause an alert saying something like 'invalid certificate' -- but NOT 'decrypt error'. 'decrypt error' in TLS has a definite meaning and it has nothing to do with RSA.

[dave_thompson_085]

@Shankar: then that's either on a different connection (or operation) or it's being ignored, because 'decrypt error' alert simply is not used for any kind of cert error ever. Moreover, CertPath>Builder<Exception 'unable to find path' cannot be caused by lack of SHA2 support; that would allow the path to be built but then give a CertPath>Validator<Exception with a very different text.

[Shankar]

@dave_thompson_085 Would you recommend something to get trace those logs on client and server side. So I can facilitate to get correct info.

[dave_thompson_085]

@Shankar: as I said, set system property javax.net.debug to the value ssl. For Jboss as I recall this should be done as -Djavax.net.debug=ssl in environment variable JAVA_OPTS which is set in some script (or Windows batch) file(s) that I don't remember exactly. For your client, you haven't told us anything, but usually that same option somehow on the command that runs the java executable.

[Shankar]

Below are the error: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at