Nginx gives "unsupported certificate purpose" for a client certificate from an iPad, but the same cert works fine on desktop browsers?

Question

Our company's web application uses client certificates to authenticate. We want to add some iPad clients to the mix for inventory counting, etc. Client certificate authentication is working just fine in desktop browsers, but when we use the exact same certificate that works in a desktop browser on an iPad, we get this error in nginx:

7200#7200: *2 client SSL certificate verify error: (26:unsupported certificate purpose) while reading client request headers

Nginx returns a 400 bad request "The SSL Certificate Error" to the iPad client.

The CA public cert and intermediate CA cert is installed on the iPad, and installed on the server as well. Again, client certificate authentication works just fine for desktop browsers with our setup.

Is this an iPad problem, an nginx problem, or a certificate problem? And how can we troubleshoot and solve it?

Updating with more information:

openssl x509 -purpose for the cert used to create the pkcs file is:

Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : Yes
S/MIME signing CA : No
S/MIME encryption : Yes
S/MIME encryption CA : No
CRL signing : Yes
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

... which appears correct.

The command used to create the pkcs file is:

openssl pkcs12 -export -out file.pk12 -inkey file.key -in file.crt -nodes -passout pass:mypassword

The iPad install profile dialog claims the Identity Certificate is not signed, but lets me install it.

Update 2

Possible clue:

When you view the certificate details on the iPad after it is installed, the iPad says "signed by" and lists the certificates own name. When you view the certificate in Firefox, Firefox shows the correct CA in the Issued By field. I'm not sure if this is an iPad glitch or the certificate isn't signed right.

Here's the exact code (with file names simplified) used to create and sign the certificates:

openssl genrsa -out file.key 4096

openssl req -new -key file.key -out file.csr -subj "$DN"

openssl x509 -req -days 365 -in file.csr -CA ca.crt -CAkey ca.key -set_serial $SerialNumber -out file.crt

openssl pkcs12 -export -out file.pk12 -inkey file.key -in file.crt -certfile ca.pem -nodes -passout pass:secret 2>&1

Notes: $DN and $SerialNumber are generated by PHP and are omitted here. ca.pem is the certificate authority's key and public cert combined into one file.

Update 3

I'm adding the output of Nginx's error log at the debug level for a request from the iPad. Company name and other sensitive information was replaced with generic words.

2017/02/19 14:17:37 [debug] 20917#20917: post event 000055D4E36D71A0
2017/02/19 14:17:37 [debug] 20917#20917: delete posted event 000055D4E36D71A0
2017/02/19 14:17:37 [debug] 20917#20917: accept on 0.0.0.0:443, ready: 1
2017/02/19 14:17:37 [debug] 20917#20917: posix_memalign: 000055D4E368EA20:512 @16
2017/02/19 14:17:37 [debug] 20917#20917: *94 accept: xx.xx.xx.xx:62856 fd:10
2017/02/19 14:17:37 [debug] 20917#20917: *94 event timer add: 10: 60000:1487531917179
2017/02/19 14:17:37 [debug] 20917#20917: *94 reusable connection: 1
2017/02/19 14:17:37 [debug] 20917#20917: *94 epoll add event: fd:10 op:1 ev:80002001
2017/02/19 14:17:37 [debug] 20917#20917: accept() not ready (11: Resource temporarily unavailable)
2017/02/19 14:17:37 [debug] 20917#20917: *94 post event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 delete posted event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 http check ssl handshake
2017/02/19 14:17:37 [debug] 20917#20917: *94 http recv(): 1
2017/02/19 14:17:37 [debug] 20917#20917: *94 https ssl handshake: 0x16
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL ALPN supported by client: spdy/3.1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL ALPN supported by client: spdy/3
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL ALPN supported by client: http/1.1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL ALPN selected: http/1.1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL server name: "our.server.com"
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_do_handshake: -1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_get_error: 2
2017/02/19 14:17:37 [debug] 20917#20917: *94 reusable connection: 0
2017/02/19 14:17:37 [debug] 20917#20917: *94 post event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 delete posted event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL handshake handler: 0
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_do_handshake: -1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_get_error: 2
2017/02/19 14:17:37 [debug] 20917#20917: *94 post event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 delete posted event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL handshake handler: 0
2017/02/19 14:17:37 [debug] 20917#20917: *94 verify:0, error:26, depth:1, subject:"/CN=OUR-COMPANY Client CA/ST=State/C=US/O=OUR-COMPANY Client CA", issuer:"/C=US/ST=State/L=City/O=OUR-COMPANY, Inc/CN=OUR-COMPANY, Inc"
2017/02/19 14:17:37 [debug] 20917#20917: *94 verify:1, error:26, depth:2, subject:"/C=US/ST=State/L=City/O=OUR-COMPANY, Inc/CN=OUR-COMPANY, Inc", issuer:"/C=US/ST=State/L=City/O=OUR-COMPANY, Inc/CN=OUR-COMPANY, Inc"
2017/02/19 14:17:37 [debug] 20917#20917: *94 verify:1, error:26, depth:1, subject:"/CN=OUR-COMPANY Client CA/ST=State/C=US/O=OUR-COMPANY Client CA", issuer:"/C=US/ST=State/L=City/O=OUR-COMPANY, Inc/CN=OUR-COMPANY, Inc"
2017/02/19 14:17:37 [debug] 20917#20917: *94 verify:1, error:26, depth:0, subject:"/C=US/ST=State/L=City/O=OUR-COMPANY Client Certificate/CN=OUR-COMPANY-MUS-58A9EEA5", issuer:"/CN=OUR-COMPANY Client CA/ST=State/C=US/O=OUR-COMPANY Client CA"
2017/02/19 14:17:37 [debug] 20917#20917: *94 ssl new session: F89EA5F8:32:1533
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_do_handshake: 1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD"
2017/02/19 14:17:37 [debug] 20917#20917: *94 reusable connection: 1
2017/02/19 14:17:37 [debug] 20917#20917: *94 http wait request handler
2017/02/19 14:17:37 [debug] 20917#20917: *94 malloc: 000055D4E3702330:1024
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_read: -1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_get_error: 2
2017/02/19 14:17:37 [debug] 20917#20917: *94 free: 000055D4E3702330
2017/02/19 14:17:37 [debug] 20917#20917: *94 post event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 delete posted event 000055D4E36D7380
2017/02/19 14:17:37 [debug] 20917#20917: *94 http wait request handler
2017/02/19 14:17:37 [debug] 20917#20917: *94 malloc: 000055D4E3702330:1024
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_read: 313
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_read: -1
2017/02/19 14:17:37 [debug] 20917#20917: *94 SSL_get_error: 2
2017/02/19 14:17:37 [debug] 20917#20917: *94 reusable connection: 0
2017/02/19 14:17:37 [debug] 20917#20917: *94 posix_memalign: 000055D4E369B8A0:4096 @16
2017/02/19 14:17:37 [debug] 20917#20917: *94 http process request line
2017/02/19 14:17:37 [debug] 20917#20917: *94 http request line: "GET / HTTP/1.1"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http uri: "/"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http args: ""
2017/02/19 14:17:37 [debug] 20917#20917: *94 http exten: ""
2017/02/19 14:17:37 [debug] 20917#20917: *94 posix_memalign: 000055D4E3703F20:4096 @16
2017/02/19 14:17:37 [debug] 20917#20917: *94 http process request header line
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "Host: our.server.com"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "Accept-Language: en-us"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "Connection: keep-alive"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "Accept-Encoding: gzip, deflate"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header: "User-Agent: Mozilla/5.0 (iPad; CPU OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13G36"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http header done
2017/02/19 14:17:37 [info] 20917#20917: *94 client SSL certificate verify error: (26:unsupported certificate purpose) while reading client request headers, client: xx.xx.xx.xx, server: our.server.com, request: "GET / HTTP/1.1", host: "our.server.com"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http finalize request: 495, "/?" a:1, c:1
2017/02/19 14:17:37 [debug] 20917#20917: *94 event timer del: 10: 1487531917179
2017/02/19 14:17:37 [debug] 20917#20917: *94 http special response: 495, "/?"
2017/02/19 14:17:37 [debug] 20917#20917: *94 http set discard body
2017/02/19 14:17:37 [debug] 20917#20917: *94 xslt filter header
2017/02/19 14:17:37 [debug] 20917#20917: *94 HTTP/1.1 400 Bad Request
Server: nginx
Date: Sun, 19 Feb 2017 19:17:37 GMT
Content-Type: text/html
Content-Length: 224
Connection: close

Solution

Possible answer: The iPad had the intermediate CA certificate (the one used to sign the client certificates) in its trusted CA store. When I deleted the intermediate CA from the iPad, the client certificate was able to work with Nginx. I also tried installing the intermediate CA as a trusted CA in desktop Firefox, and as soon as I did, I got the same error from Nginx in the Firefox client.

So while certificates are now working, I'm stumped as to why trusting the CA that issued the client certs would break them.

Since I still have a bounty to award, I'll give the bounty to whomever can either:

1. Explain why this is the normal/expected behavior with supporting documentation.

-or-

2. If this is not normal, tell me what is mis-configured to cause such bizarre behavior.

Comments

[Patrick Mevzek]

Are you 100% sure the iPad client really sends the certificate you think it should send? Do some network captures to verify that or add more logging on the server side to see what the client sends.

[Nick]

The iPad prompts for a certificate and I can choose one. I tried increasing the Nginx log level, but it still doesn't include headers, and I couldn't figure out how to get the request headers into the Nginx error or access logs.

[Nick]

I was able to log $ssl_client_cert and I can see that the client cert is coming in to Nginx from the iPad.

[Matthew Ife]

I'd suggest getting a packet capture of a working and non-working SSL negotiation. This occurs at the handshake end. I'm wondering if its something to do with the ipad not sending intermediate certs or something like that.

[Nick]

I'm not sure how to get a packet capture. Is there a software that I can install on the server to do this? The server is Ubuntu 16.04.

[Matthew Ife]

@Nick you can run tcpdump -s512 -ni ethX tcp and host <clientip> and port 443

[Nick]

These same client certificates work just fine from desktop Firefox.

[GreyWolf]

You can run nginx with error_log /path/to/log debug; and it will tell you, what exactly it does not like.

[Nick]

I added the debug level output to the question above. I'm still not seeing anything beyond the SSL certificate verify error: (26:unsupported certificate purpose) in the original question.

[GreyWolf]

Can you put client certificate itself (without key), or output of openssl x509 -in cert.crt -text somewhere to look at?

[Matthew Ife]

Can you supply the intermediate certificate details? Perhaps theres a cert usage flag in it that doesn't permit it to be used for signing other certificates?