Apache certificate chain is not being sent

Question

Good day!

I've searched google high and low. Goodness, I've never gone past page 2 of search results before! But this one stumps me.

I've got a website that I am securing with a certificate signed by an intermediary certificate, signed by a self-signed CA certificate. Every website stackexchange q/a, etc seems to indicate the following:

1. Concatenate your server certificate with the intermediary certificate, then with your CA certificate. (i.e. cat apache.pem > chain_file; cat intermediary.pem >> chain_file; cat ca.pem >> chain_file)
2. Point to this new file on your apache ssl config using SSLCertificateFile since SSLCertificateChainFile is being deprecated.
3. Check with a browser (or ssllabs.com or sslshopper.com) to see if the certificate is being exported.

However, when I check with a browser, I see only the server certificate and does not contain the intermediary or CA certificates. The same is confirmed with both ssllabs and sslshopper.

Is there an option I'm missing that ensures that you're publishing the entire chain and not just the server's certificate?

Thanks

Solution

If you have OpenSSL at your disposal, verifying this is exceedingly easy:

cat /dev/null | openssl s_client -showcerts -servername example.com -connect example.com:443

I’ll explain the components:

• cat /dev/null: openssl s_client works much like telnet. It establishes the connection and you can then interact with it. When openssl encounters EOF in the input stream, the connection is closed. cat /dev/null immediately results in EOF. Alternatively, you can use CtrlD to issue EOF on the terminal.
• openssl s_client: “SSL/TLS client program”
• -showcerts: Show certificate blobs in PEM format. Only shows server certificate otherwise.
• -servername example.com: Set server name for SNI. Required when doing HTTPS name-based vhosts.
• -connect example.com:443: Connect to example.com at port 443, the standard HTTPS port.

This results in something like that:

$ cat /dev/null | openssl s_client -showcerts -servername inbox.google.com -connect inbox.google.com:443
CONNECTED(00000003)
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = mail.google.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
-----BEGIN CERTIFICATE-----
snip
-----END CERTIFICATE-----
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
snip
-----END CERTIFICATE-----
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
-----BEGIN CERTIFICATE-----
snip
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3767 bytes and written 469 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 1B47CE2ADB10CE410C8048C3AAEF7CEF1B2B76C6D2DF5EDE78FE015A6DA44207
    Session-ID-ctx:
    Master-Key: E9AE458F6D72D507F422DA2340C7345AC6EDB087278E62A5FDA754897EC6BDF5C336AFBF6B88554E358C675A3545B724
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 100800 (seconds)
    TLS session ticket:
    snip

    Start Time: 1460049698
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
DONE

I used a Google domain here because it has a deeper certificate chain than example.com. The certificate here has “Subject Alternate Names” and as such is also valid for inbox.google.com.

An incorrectly configured server would skip the intermediary certificate, called “Google Internet Authority G2” here. The Equifax certificate is sort-of redundant because GeoTrust is an established CA that’s directly trusted in your browser.

Comments

[Daniel B]

You can’t really see that in your browser. Did you try one of the sites indicated?

[user1686]

Instructions look fine. Make sure the certificates themselves are correct though. (Maybe your cert was issued by a different intermediate than you're trying to bundle?) Try also ssl-tools.net to check the chain.

[scuba_mike]

DanielB, yes. The same has been confirmed by both ssllabs.com and sslshopper.com grawity, that was my thinking too but I have checked the Subject and Issuer.

[scuba_mike]

Additionally, I ran the following command: openssl verify -CAfile ca.pem -untrusted intermediary.pem apache.pem and this return OK.

[Daniel B]

That doesn’t test your server configuration in any way, though. Since you have OpenSSL available, things are quite easy. I’ll write an answer soon.

[scuba_mike]

Thanks for the notes. I was not aware of the servername option. It wasn't in the man page. Anyway, running it gives me "gethostbyname failure connect:errno=0" The difference between google and me is that google is using an explicit CN (e.g. mail.google.com) and I'm using a wildcard certificate. Do I need to have an explicit server cert?

[Daniel B]

No. The error you’re getting indicates the destination host (-connect argument) cannot be resolved. It is entirely unrelated to TLS, certificates and whatnot. Are you sure you didn’t have a typo in the command line?

[scuba_mike]

Hmm...I didn't have a typo before but it seems to be working now (s_client). It shows 3 certs but they are have a depth of 0 and the same certificate information, in this case the certificate for the server. But each line has an error. First error: verify error:num=20:unable to get local issuer certificate'. Second error: verify error:num=27:certificate not trusted` Third error: verify error:num=21:unable to verify the first certificate If you'd like to try, feel free: aws.mikesoh.com

[Daniel B]

Yeah, I found your mistake. It’s yet another case of RTFM. ;) You have Apache 2.4.7, but the new version of the SSLCertificateFile directive that supports intermediary (or, more generally: multiple) certificates is only available starting from 2.4.8.

[scuba_mike]

FRAK! Ok now apache won't start. It's giving me an error X509_check_private_key:key values mismatch. checking the web for this now...

[scuba_mike]

I just ran openssl s_client -showcerts -servername aws.mikesoh.com -connect aws.mikesoh.com:443 with the change in the config file to use SSLCertificateChainFile instead. I'm still getting the same three certificate and a depth=0.

[Daniel B]

Did you verify the file actually contains the correct certificates in the correct order? Because it looks like there might be duplicates. I'll look into it some more tomorrow.