Two Way SSL Error - 400 The SSL certificate error just for client certificate

ssl ssl-certificate ssl-certificate-errors

47,561

Solution 1

Finally, I have pinned down the root cause of the problem. There were two problems with my setup.

a) For two-way SSL, the certificate signed by the Intermediate CA must have clientAuth in extendedKeyUsage (Thanks to @dave_thompson_085) which can be verified by the below command

$ openssl x509 -in /path/to/client/cert -noout -purpose | grep 'SSL client :'
SSL client : Yes

b) Another, thing which was missing was ssl_verify_depth parameter in the nginx config file which must be 2 or more. It does not make much sense to make the number bigger than 2 in my case, but it works with any number other than 1 (which is default value). Interestingly, this is not required in nginx v1.12.X (my colleague with the exact same setup didn't have to specify this). However, it didn't work for me (nginx v1.13.5) until I used this parameter.

I can have a sound sleep after 3 days of headbanging.

TIP: Don't depend on curl much to troubleshoot two-way SSL issues, try openssl s_client instead. curl can give misleading results sometimes, see this. I too fumbled around for a while in my Ubuntu 16.04 docker container.

Solution 2

openssl s_client -cert $file can provide only 'the' client cert (singular, one), not a chain; it can optionally provide the privatekey as well, if you don't specify -key, but still not any cert(s) other than the client cert. If you put any additional cert(s) in that file they are totally and completely ignored.

But libssl uses certs from the truststore to fill out the client's cert chain in addition to its nominal purpose of validating the server's cert chain. Read the subtly definitive words of the man page:

-CApath directory

The directory to use for server certificate verification. This directory must be in "hash format", see verify for more information. These are also used when building the client certificate chain.

-CAfile file

A file containing trusted certificates to use during server authentication and to use when attempting to build the client certificate chain.

Since you need to specify your private root to verify the server chain, if you instead give -CAfile a file containing both your root and your intermediate then both directions will work.

Unlike some other implementations where putting an intermediate in the truststore causes validation to stop at that point, OpenSSL will use truststore certs to complete a received chain that is incomplete, but by default still validates up to a root from the truststore; to stop earlier you must specify -partial_chain (new in 1.1.0).

However, I'm surprised configuring the server with ssl_client_certificate containing root and intermediate didn't work. nginx also uses OpenSSL and as I said libssl will use intermediate(s) from truststore to complete (though not validate) a received chain. So that approach, although officially not standard-conforming (the sender is responsible for sending a full chain modulo root) should actually work. I'll try to test later when I have a chance.

47,561

vikas027

Updated on September 18, 2022

Comments

vikas027 over 1 year

I am trying to configure two-way SSL with SSL certs (for server and client) signed by Intermediate CAs. This is what I have done so far following this tutorial.

Server - nginx application

Nginx is configured with SSL certificate (signed by an Intermediate CA).

server {
  listen 443;
  server_name app-ca.test.com;
  ssl on;

  ssl_certificate /root/ca/intermediate/certs/app-plus-intermediate.pem;
  ssl_certificate_key /root/ca/intermediate/private/app-ca-interm-ca.test.com.key.pem;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers  HIGH:!aNULL:!MD5;
  ssl_prefer_server_ciphers   on;

  # I have also tried adding the Intermediate CA cert in vain
  # ssl_client_certificate /root/client_rootca_intermediate.crt;
  ssl_client_certificate /root/client_rootca.crt;
  ssl_verify_client on;

  location / {
      root /usr/share/nginx/massl;
      index index.html index.htm;
  }
}

Client - curl or OpenSSL s_client

I have a client certificate signed by some other Intermediate CA, which fails with 400 The SSL certificate error

I have also tried to pass (-cert option in openssl command) Client's Intermediate CA and Root CA along with the client certificate in vain.

$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem > /root/ca/intermediate/certs/client_plus_intermediate.cert.pem

$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem > /root/ca/intermediate/certs/intermediate_plus_client.cert.pem

$ cat /root/ca/intermediate/certs/client.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem /root/ca/certs/ca.cert.pem > /root/ca/intermediate/certs/client_plus_intermediate_plus_ca.cert.pem

Short Error Logs

<html>
<head><title>400 The SSL certificate error</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.13.5</center>
</body>
</html>

Long Error Logs

I have shortened the longs hashes for brevity.

$ openssl s_client -connect app-ca.test.com:443 -tls1 -key /root/ca/intermediate/private/client.key.pem -cert /root/ca/intermediate/certs/client.cert.pem -CAfile /root/server_rootca.crt -state -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x2239a90 [0x226e3c3] (181 bytes => 181 (0xB5))
0000 - 16 03 01 00 b0 01 00 00-ac 03 01 16 ed fa 81 3e   ...............>
0010 - fc 25 c1 55 73 8a ca 5f-d3 56 11 a6 0f 38 6e 3c   .%.Us.._.V...8n<
0020 - 52 fb 1f 9b fb 4f 4f 3e-5a fb 82 00 00 64 c0 14   R....OO>Z....d..
0090 - 00 ff 01 00 00 1f 00 0b-00 04 03 00 01 02 00 0a   ................
00a0 - 00 0a 00 08 00 17 00 19-00 18 00 16 00 23 00 00   .............#..
00b0 - 00 0f 00 01 01                                    .....
SSL_connect:SSLv3 write client hello A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 42                                    ....B
read from 0x2239a90 [0x2269e78] (66 bytes => 66 (0x42))
0000 - 02 00 00 3e 03 01 6f e5-89 1d bd 5a 58 26 d7 11   ...>..o....ZX&..
0010 - 8a 05 fd 2a 04 96 58 2e-2e 19 a7 89 46 a0 5b 21   ...*..X.....F.[!
0020 - c3 90 1c 3e 0b e6 00 c0-14 00 00 16 ff 01 00 01   ...>............
0030 - 00 00 0b 00 04 03 00 01-02 00 23 00 00 00 0f 00   ..........#.....
0040 - 01 01                                             ..
SSL_connect:SSLv3 read server hello A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 0c ab                                    .....
read from 0x2239a90 [0x2269e78] (3243 bytes => 3243 (0xCAB))
0000 - 0b 00 0c a7 00 0c a4 00-06 64 30 82 06 60 30 82   .........d0..`0.

0c90 - 5f b6 c7 86 5d 41 b3 fb-9c fe d3 0a 26 01 f9 d9   _...]A......&...
0ca0 - a6 ae 7f ff 4f c7 0b e8-97 b3 1c                  ....O......
depth=2 C = GB, ST = England, L = Melbourne, O = Alice Ltd, OU = IT Services, CN = server-and-ca.test.com, emailAddress = [email protected]
verify return:1
depth=1 C = GB, ST = England, O = Alice Ltd, OU = Shared Services, CN = server-and-interm-ca.test.com, emailAddress = [email protected]
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Alice Ltd, OU = Alice Ltd Web Services, CN = app-ca-interm-ca.test.com, emailAddress = [email protected]
verify return:1
SSL_connect:SSLv3 read server certificate A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 01 4b                                    ....K
read from 0x2239a90 [0x2269e78] (331 bytes => 331 (0x14B))
0000 - 0c 00 01 47 03 00 17 41-04 13 5d 81 04 36 18 e7   ...G...A..]..6..
0010 - da bf 5e 30 dd d8 ee 77-f9 56 aa 77 8b 9e cd 3e   ..^0...w.V.w...>.
0110 - d1 82 65 0f 5d 9c 03 ba-5f 7f 62 33 a8 a6 62 8e   ..e.]..._.b3..b.
0120 - f2 5c 03 1d 4d 47 04 16-cb 80 09 39 32 be ca 23   .\..MG.....92..#
0130 - 41 95 36 a6 4b 6b f0 6c-df a5 4b 26 d4 4a c5 f3   A.6.Kk.l..K&.J..
0140 - 99 0d c8 d8 aa 5d f8 88-86 b3 15                  .....].....
SSL_connect:SSLv3 read server key exchange A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 bc                                    .....
read from 0x2239a90 [0x2269e78] (188 bytes => 188 (0xBC))
0000 - 0d 00 00 b4 03 01 02 40-00 ae 00 ac 30 81 a9 31   [email protected]
0010 - 0b 30 09 06 03 55 04 06-13 02 47 42 31 10 30 0e   .0...U....GB1.0.
0090 - 06 09 2a 86 48 86 f7 0d-01 09 01 16 1b 72 6f 6f   ..*.H........roo
00a0 - 74 40 63 6c 69 65 6e 74-2d 61 6e 64 2d 63 61 2e   t@client-and-ca.
00b0 - 74 65 73 74 2e 63 6f 6d-0e                        test.com.
00bc - <SPACES/NULS>
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
write to 0x2239a90 [0x2273910] (1593 bytes => 1593 (0x639))
0000 - 16 03 01 06 34 0b 00 06-30 00 06 2d 00 06 2a 30   ....4...0..-..*0
0010 - 82 06 26 30 82 04 0e a0-03 02 01 02 02 02 10 00   ..&0............
05f0 - 29 2a 6c 40 d1 ed 8f 6d-15 b2 cd 6a 7b 72 30 91   )*[email protected]{r0.
0600 - ea 29 16 48 f2 11 21 15-3a 50 32 8b 95 87 b8 09   .).H..!.:P2.....
0610 - 11 84 9a a4 d2 b8 46 33-7a a2 79 51 ba 23 8c 96   ......F3z.yQ.#..
0620 - 45 62 2e b9 f5 ea 23 79-53 e0 cb 72 1f e6 19 d4   Eb....#yS..r....
0630 - 75 18 a8 2e 44 2f f3 8b-a7                        u...D/...
SSL_connect:SSLv3 write client certificate A
write to 0x2239a90 [0x2273910] (75 bytes => 75 (0x4B))
0000 - 16 03 01 00 46 10 00 00-42 41 04 b9 b3 02 d2 bc   ....F...BA......
0010 - e2 8b 49 a7 f6 8c 59 66-fc 0e 39 79 c7 23 34 e9   ..I...Yf..9y.#4.
0020 - 3e 04 98 3a 60 78 1d aa-51 06 46 80 09 10 c4 7e   >..:`x..Q.F....~
0030 - a5 e7 05 d1 82 f2 0d bb-9a ca e7 29 01 0b 88 6d   ...........)...m
0040 - ed c3 52 73 b1 d4 3a 95-00 e8                     ..Rs..:...
004b - <SPACES/NULS>
SSL_connect:SSLv3 write client key exchange A
write to 0x2239a90 [0x2273910] (267 bytes => 267 (0x10B))
0000 - 16 03 01 01 06 0f 00 01-02 01 00 5e 29 8e 7c 69   ...........^).|i
0010 - 1e 10 0d 01 39 35 db 18-7e 4a a7 12 ae 12 7e f0   ....95..~J....~.
0020 - d6 93 c5 0a ba 5d e4 f1-a4 ae 8f c4 7d 52 80 16   .....]......}R..
00f0 - 6f 1f 56 73 bc ab 7f 07-1d f7 b4 ec d7 58 57 cd   o.Vs.........XW.
0100 - cd e0 37 b3 58 09 3a 75-93 02 ab                  ..7.X.:u...
SSL_connect:SSLv3 write certificate verify A
write to 0x2239a90 [0x2273910] (6 bytes => 6 (0x6))
0000 - 14 03 01 00 01 01                                 ......
SSL_connect:SSLv3 write change cipher spec A
write to 0x2239a90 [0x2273910] (53 bytes => 53 (0x35))
0000 - 16 03 01 00 30 24 90 78-08 d3 10 f3 f8 e3 c8 86   ....0$.x........
0010 - 82 f1 54 d1 38 7b 57 7b-83 a3 49 b9 3b 80 b2 86   ..T.8{W{..I.;...
0020 - 54 74 92 ec 9a a7 e7 28-1a ec 72 4c 64 8e f3 e3   Tt.....(..rLd...
0030 - 08 96 89 2a 03                                    ...*.
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 06 ea                                    .....
read from 0x2239a90 [0x2269e78] (1770 bytes => 1770 (0x6EA))
0000 - 04 00 06 e6 00 00 01 2c-06 e0 09 8d 58 07 45 c9   .......,....X.E.
0010 - 58 49 42 f4 13 00 47 12-be 22 a2 e3 a0 b6 22 bd   XIB...G.."....".
06d0 - a1 11 26 db 43 c8 6e 47-2f 40 65 61 e1 4e ef 0a   ..&.C.nG/@ea.N..
06e0 - 57 e0 28 19 2d 0d c6 7f-ae 2e                     W.(.-.....
SSL_connect:SSLv3 read server session ticket A
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 14 03 01 00 01                                    .....
read from 0x2239a90 [0x2269e78] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 30                                    ....0
read from 0x2239a90 [0x2269e78] (48 bytes => 48 (0x30))
0000 - 7d 5f 53 a4 5e 85 67 67-8d 6c d6 6e 93 cd c6 75   }_S.^.gg.l.n...u
0010 - c1 83 17 d9 a8 e3 89 23-86 6b 8a 04 2d 46 7e 95   .......#.k..-F~.
0020 - 15 46 a4 ec 73 f3 3d 78-1b 0e 94 62 79 cf 96 3d   .F..s.=x...by..=
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Alice Ltd/OU=Alice Ltd Web Services/CN=app-ca-interm-ca.test.com/[email protected]
   i:/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
 1 s:/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
   i:/C=GB/ST=England/L=Melbourne/O=Alice Ltd/OU=IT Services/CN=server-and-ca.test.com/[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGYDCCBEigAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgagxCzAJBgNVBAYTAkdC
MRAwDgYDVQQIDAdFbmdsYW5kMRIwEAYDVQQKDAlBbGljZSBMdGQxGDAWBgNVBAsM
zBcik+fj+MUtDzhEl6EuW1ILjAvt5u4KBxj6d0yAXzleACOYncYWWzMfQdrFmwKh
W2opZQ==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Alice Ltd/OU=Alice Ltd Web Services/CN=app-ca-interm-ca.test.com/[email protected]
issuer=/C=GB/ST=England/O=Alice Ltd/OU=Shared Services/CN=server-and-interm-ca.test.com/[email protected]
---
Acceptable client certificate CA names
/C=GB/ST=England/L=Sydney/O=Something/OU=Shared Services/CN=client-and-ca.test.com/[email protected]
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5682 bytes and written 2175 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 2AF7BFD60D3EC4686EAAAE1971FBD8999E65C5C80A32182CB9A668B1411DB09C
    Session-ID-ctx:
    Master-Key: B3F714B4ACB61C6310311025B25AFBAFA9E9AAEBB5ACD5FEEAE5DCAE2690DECBFA4EC5CBD2C8A50F349F43026CD0C564
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 09 8d 58 07 45 c9 58 49-42 f4 13 00 47 12 be 22   ..X.E.XIB...G.."
    0010 - a2 e3 a0 b6 22 bd 0d 71-c9 46 bd ab 84 85 06 f7   ...."..q.F......
    06b0 - 66 76 1f 3e 49 23 dc 2b-be 9e d5 03 b8 a5 a1 7d   fv.>I#.+.......}
    06c0 - 4d 56 79 3f 81 78 a1 11-26 db 43 c8 6e 47 2f 40   MVy?.x..&.C.nG/@
    06d0 - 65 61 e1 4e ef 0a 57 e0-28 19 2d 0d c6 7f ae 2e   ea.N..W.(.-.....

    Start Time: 1506251677
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
GET / HTTP/1.0
write to 0x2239a90 [0x226e3c6] (90 bytes => 90 (0x5A))
0000 - 17 03 01 00 20 ca 44 95-8c a0 32 52 4d da d8 02   .... .D...2RM...
0010 - db bd 97 88 0e e3 cb b9-9e fb 50 7e 71 24 37 83   ..........P~q$7.
0020 - f8 48 03 a0 a1 17 03 01-00 30 db 99 b2 0c 6c e6   .H.......0....l.
0030 - f4 25 3d 54 2f b1 a3 3c-be 2a 36 94 6c ce 6d 8d   .%=T/..<.*6.l.m.
0040 - 3d 54 82 d3 f0 2a 40 3d-fc 3f 1b 3e 4a 40 10 e5   =T...*@=.?.>J@..
0050 - 1d eb ab 00 69 f1 e0 4a-27 47                     ....i..J'G

write to 0x2239a90 [0x226e3c6] (74 bytes => 74 (0x4A))
0000 - 17 03 01 00 20 95 06 3d-51 d5 7c c2 05 ef a7 d6   .... ..=Q.|.....
0010 - 2b 25 9c dd ec 5f 7c c0-15 83 c6 ca ea 47 a1 b2   +%..._|......G..
0020 - 82 2d 46 7d 64 17 03 01-00 20 3b 2e 36 63 10 b3   .-F}d.... ;.6c..
0030 - 50 c7 ec 36 a4 27 a0 4d-db bb 83 b5 c6 e8 d5 fa   P..6.'.M........
0040 - ca 76 dc e7 63 8f 94 b3-24 3f                     .v..c...$?
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 17 03 01 01 a0                                    .....
read from 0x2239a90 [0x2269e78] (416 bytes => 416 (0x1A0))
0000 - a6 8b c1 bb a4 aa 12 2e-81 d9 45 41 74 0e 33 a4   ..........EAt.3.
0190 - 37 be 58 ca 01 80 fc 7c-79 2b 3f 54 a4 cd 4a 07   7.X....|y+?T..J.
HTTP/1.1 400 Bad Request
Server: nginx/1.13.5
Date: Sun, 24 Sep 2017 11:14:49 GMT
Content-Type: text/html
Content-Length: 231
Connection: close

<html>
<head><title>400 The SSL certificate error</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.13.5</center>
</body>
</html>
read from 0x2239a90 [0x2269e73] (5 bytes => 5 (0x5))
0000 - 15 03 01                                          ...
0005 - <SPACES/NULS>
read from 0x2239a90 [0x2269e78] (32 bytes => 32 (0x20))
0000 - c3 75 ba 40 21 83 f7 0e-11 98 7b 44 84 bb 23 d5   .u.@!.....{D..#.
0010 - 80 32 1e 3e b6 b7 dd 4a-16 09 31 e9 62 a9 cd a3   .2.>...J..1.b...
SSL3 alert read:warning:close notify
closed
write to 0x2239a90 [0x226e3c3] (37 bytes => 37 (0x25))
0000 - 15 03 01 00 20 bd 18 f2-df 1b 84 fc 8e e0 80 a1   .... ...........
0010 - 2f 6f 31 b4 4c fc 1c e5-36 1f c5 fb 5d c0 f8 dc   /o1.L...6...]...
0020 - 19 6b 03 c3 2d                                    .k..-
SSL3 alert write:warning:close notify

Interestingly, the above command works fine if I use certificates of Client's Root CA or Client's Intermediate CA.

Steffen Ullrich over 6 years

The client has to send not only the leaf certificate but also all intermediate certificates needed to build the trust chain to the trusted root at the server. Alternatively the server has to trust not only the root but also needs to know these intermediates already.
vikas027 over 6 years

@SteffenUllrich Yeah, that is my understanding as well. I did try to pass all certificates of the client in vain. I have updated my question with what I have tried. I am missing something here, can't really figure out what. :(
dave_thompson_085 over 6 years

PS: what you shortened are dumps of the messages, not hashes.

vikas027 over 6 years

Sorry, the second server block does not make much sense for two-way SSL. I think the problem you are trying to address is basic SSL (one way SSL), which is not a problem for me. My SSL certificate file (/root/ca/intermediate/certs/app-plus-intermediate.pem) is not a wildcard certificate and has application certificate plus the certificate signed by Intermediate CA.
vikas027 over 6 years

Thanks, for the detailed explanation. I have tried to add chain (Intermediate CA plus Root CA) with -CAfile too, but still, have the same 400 error. openssl -connect server.test.com:443 -tls1 -key /root/ca/intermediate/private/client.key.pem -cert /root/ca/intermediate/certs/client.cert.pem -CAfile /root/ca/intermediate/certs/ca-chain.cert.pem Interestingly, now I have a different error in nginx error log file SSL certificate verify error: (26:unsupported certificate purpose) while reading client request headers I think I will try to use -CApath now.
dave_thompson_085 over 6 years

@vikas027: that different error is significant; 'unsupported purpose' (X509V3_R_INVALID_PURPOSE in the code) means it found the full chain (as desired) and started the validation. I don't know what validation options nginx uses, but most likely this is a problem with BasicConstraints, KeyUsage, or ExtendedKeyUsage; try looking at those and make sure (if present) they're correct. CApath (used correctly) won't achieve anything different than CAfile -- both of them are part of local truststore, although the file organization is different.
Zeeshan over 3 years

Main highlight here: clientAuth in extendedKeyUsage
Zeeshan over 3 years

Thanks to your answer, I had only 3 minutes of head banging. Just added clientAuth in extendedKeyUsage for the certificate used on my client machine and it was sorted.