SSL Root CA certificate is not recognized, although present in the trust store. Why?

Question

Background:

• Ubuntu Server 14.10 64-bit on aws.amazon.com/ec2
• Cheap PositiveSSL server certificate from COMODO
• 1 server certificate, 2 intermediate CA certificates and 1 Root CA certificate as ZIP archive from COMODO
• Citadel's WebCit httpsd

Problem:

The concatenated certificate chain seems to be correct but verification fails.

openssl s_client myhost:port

shows the certificate chain and the issuer-subject pairs line up correctly through the chain, but:

verify error:num=19:self signed certificate in certificate chain

The root CA certificate is not accepted by openssl, although it is found per default in the Ubuntu server trust store.

Specifically: AddTrustExternalCARoot.crt received per email from COMODO and /etc/ssl/certs/AddTrust_External_Root.pem which links to /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt are indentical.

What is wrong here?

Solution 1

OpenSSL at least through current (1.0.2a) has a bug where s_client with NO -CA{path,file} argument doesn't actually use the default truststore as it should, and thus fails to verify certs that are valid according to that truststore. (Also s_server and s_time, but caring about verification in those is rare.) See https://serverfault.com/questions/607233/how-to-make-openssl-s-client-using-default-ca . A fix is announced in dev, but may take some time to be released and distributed. In the meantime you need to explicitly specify the -CA* argument(s). Note that openssl verify does not have this bug, and therefore correctly reported the cert/chain as valid.

UPDATES 2015/08/26: fix was released 2015/06/12 in 1.0.1o and 1.0.2c. Also, while investigating something else I found that RedHat packages may have been okay. More specifically the CentOS source RPM for openssl-1.0.1e-30.el6.11 which I understand is a copy of the RedHat one (but can't easily confirm) contains openssl-1.0.1c-default-paths.patch which contains changes to s_client.c s_server.c s_time.c dated 2012/12/06 that appear equivalent to (though not textually the same as) the 2015/06/12 upstream fixes. Assuming this patch was applied in RedHat and CentOS packages, which I can't easily go back and check, they would (have) work(ed) as expected.

Solution 2

I faced a similar issue with Comodo certificates recently when developing a script using Ruby. In the end it was that OpenSSL did not have it in the store, even though it looked like it did.

To test this, download all of the Comodo intermediate certs and create a cert bundle something like this (you'll need to use different cert names depending on what you downloaded):

cat EssentialSSLCA_2.crt ComodoUTNSGCCA.crt UTNAddTrustSGCCA.crt AddTrustExternalCARoot.crt > yourDomain.ca-bundle

Comodo has an article on how to do this.

Once done, try verifying the certificate again using OpenSSL and specifying the cert store on the command line:

openssl verify -untrusted yourDomain.ca-bundle cert.pem

That example was adapted from this Unix and Linux StackExchange article.

Once you've determined which certificate it is, it should be possible to add the certificate to the local cert store, which is detailed here for Ubuntu, and is something like:

Create a directory for extra CA certificates in /usr/share/ca-certificates

sudo mkdir /usr/share/ca-certificates/extra

Copy the '.crt' file to the directory

sudo cp foo.crt /usr/share/ca-certificates/extra/foo.crt

Let Ubuntu add the '.crt' file's path relative to /usr/share/ca-certificates to /etc/ca-certificates.conf

sudo dpkg-reconfigure ca-certificates

Comments

[John Siu]

If you use firefox or chrome to connect to your site, do you get any certificate warning?

[Reinhard Seifert]

Not firefox. I am using Citadel.org as an email server and the SSL certificates are for SMTPS on ports 465 and 587. Indeed Thunderbird does not complain.

[Reinhard Seifert]

The Firefox issue triggered my post here. Oh man, I am tired. So, Thunderbird does not complain on SMTPS, but Firefox does not display the HTML page and I am guessing it it the certificate, because when using openssl s_client the html of the page shows up and w3m also displays the page after complaining about the untrusted certificate. But Firefox simply does not show the page, without any warning about the certificate.

[John Siu]

Try the browser test with another device. Also use latest version if possible. I am starting to guess that the computer used for the test need root CA package update.

[Reinhard Seifert]

That is four years ago and the certificates have been revoked and replaced. On top the root CA is not from COMODO but from AddTrust; the root CA certificate is verified OK when using openssl verify AddTrustExternalCARoot.crt directly, but it is not accepted as the last certificate in a chain.

[John Siu]

This is my guess too, that's why I ask OP to test with a Chrome or Firefox browser.

[jww]

"In the end it was that OpenSSL did not have it in the store..." - OpenSSL does not have a store. You have to explicitly tell it what to use as a trust anchor. OpenSSL is not like browsers which trust nearly anything that's thrown at them.

[Reinhard Seifert]

Thanks for your immediate answers and comments guys. I will look closer at it tomorrow. I get the point. But: When doing openssl verify AddTrustExternalRootCA.crt (which is the culprit at the end of the chain) for the root certificate alone, then it passes "OK", i.e. openssl is very well aware of the certificates linked in /etc/ssl/certs.

[jotap]

So @ReinhardSeifert just to clarify the cert with the problem verifies when you run that command? If that is the case adding all the certs in the bundle should sort it out, even if it looks like they are already there.

[Reinhard Seifert]

@jotap: Yes, that is correct. Using the -CAfile option with the intermediate certs the verify of the single server certificate is "OK". Also Firefox and Thunderbird accept the chained SSL certificate - that is sortet out now. But: openssl s_client complains about the chain.

[Reinhard Seifert]

By the way the site is up now. You can try youself pointing your browser at das.email - which is ok and openssl s_client -connect das.email:443 which throws error 19.

[dave_thompson_085]

@jww OpenSSL library has a default store location, but uses it only if the application calls SSL_CTX_set_default_verify_paths() or some related X509_STORE routines. OpenSSL commandline is supposed to use the default store, but currently fails to do so for some functions including s_client due to a bug; see my answer. OTOH verify does use it, which probably added to the confusion here.

[jww]

@dave - "OpenSSL commandline is supposed to use the default store..." - that's interesting. I always thought it was by design.

[Reinhard Seifert]

Thank you. That clarified the matter and indeed it was for once not me messing things up... in the browser and MUA the chain is accepted.