How to fix certificate chain with letsencrypt / certbot?

Question

I cannot wrap my head around the following problem. Verifying the certificates of the server with openssl fails, the chain is imcomplete.

Disclaimer: I am not an admin and did not work much with certificates yet.

Verifiy with OpenSSL

$ openssl verify -CAfile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem  /etc/letsencrypt/live/co2-avatar.com/cert.pem

# /etc/letsencrypt/live/co2-avatar.com/cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
# error 2 at 2 depth lookup:unable to get issuer certificate

Check for one of the domains in the certificate

openssl s_client -connect co2avatar.org:443 -servername co2avatar.org
# CONNECTED(00000003)
# depth=0 CN = gitlab.sustainable-data-platform.org
# verify error:num=20:unable to get local issuer certificate
# verify return:1
# depth=0 CN = gitlab.sustainable-data-platform.org
# verify error:num=21:unable to verify the first certificate
# verify return:1
# ---
# Certificate chain
#  0 s:CN = gitlab.sustainable-data-platform.org
#    i:C = US, O = Let's Encrypt, CN = R3
# ---
# Server certificate
# -----BEGIN CERTIFICATE-----

Or run

curl -v https://co2avatar.org
# *   Trying 85.214.38.88:443...
# * TCP_NODELAY set
# * Connected to co2avatar.org (85.214.38.88) port 443 (#0)
# * ALPN, offering h2
# * ALPN, offering http/1.1
# * successfully set certificate verify locations:
# *   CAfile: /etc/ssl/certs/ca-certificates.crt
#   CApath: /etc/ssl/certs
# * TLSv1.3 (OUT), TLS handshake, Client hello (1):
# * TLSv1.3 (IN), TLS handshake, Server hello (2):
# * TLSv1.2 (IN), TLS handshake, Certificate (11):
# * TLSv1.2 (OUT), TLS alert, unknown CA (560):
# * SSL certificate problem: unable to get local issuer certificate
# * Closing connection 0
# curl: (60) SSL certificate problem: unable to get local issuer certificate

There might be both, a wrong configuration in my Apache VHost for the domain as well as a problem in the certificate chain itself. How can I check the last one (I've googled a lot, but most hits are about openssl verify with -CAfile or about different cert issuer)?

Do I need to check the root certifate bundle and how exactly?

Is there something like an -addtrust flag for certbot certonly?

Solution

Try openssl s_client and let you show the certs. The command is:

$ openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts

You will find that your server returns a certificate for CN = gitlab.sustainable-data-platform.org and a subject alternative name which includes your domain DNS:co2-avatar.com. So the certificate itself is fine.

If you want to combine everything into one pipeline of commands to see the content of your certificate:

echo | openssl s_client -connect co2avatar.org:443 -servername co2avatar.org -showcerts 2>/dev/null |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -noout -text

Whats missing is the intermediate certificate. This should be sent by the server as well, but the first command shows you that it is not there - only the certificate is sent by your server.

So the failing openssl is correct, as indeed the intermediate certificate is missing.

So to solve it, you need to tweak your apache configuration. This is how your configuration could look like:

Filename should be similar to /etc/apache2/sites-enabled/co2-avatar.com-le-ssl.conf

<IfModule mod_ssl.c>
SSLStaplingCache shmcb:/var/run/apache2/stapling_cache(128000)
<VirtualHost *:443>
        ServerName co2-avatar.com
        ServerAlias www.co2-avatar.com
#... 
#... insert your other stuff here...
#...

SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/co2-avatar.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLUseStapling on
</VirtualHost>
</IfModule>

Based on your description, my best guess is that the following line is wrong in your config: SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/cert.pem. It should be replaced with SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem, in order to send also the intermediate(s).

Solution UPDATE (after discussion)

It turned out in discussion that the openssl and Apache version used on this CentOS server is just older, so some features are unsupported. (Apache 2.4.6, OpenSSL 1.0.2k, intermediate configuration, no HSTS, no OCSP)

According to Mozilla SSL Configuration Generator the following generic configuration could be used in this case:

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile      /path/to/signed_certificate
    SSLCertificateChainFile /path/to/intermediate_certificate
    SSLCertificateKeyFile   /path/to/private_key
</VirtualHost>

Translated to this specific case, a resulting working config would be as the following:

<VirtualHost *:443>
    ServerName  sustainable-data-platform.org
    ServerAlias co2-avatar.com
    ServerAlias ... <include all other SAN names here>
    
    SSLEngine on
    SSLCertificateFile      /etc/letsencrypt/live/co2-avatar.com/cert.pem
    SSLCertificateChainFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem
    SSLCertificateKeyFile   /etc/letsencrypt/live/co2-avatar.com/privkey.pem

</VirtualHost>

As note for such old installations

Cross-Signed Let’s Encrypt R3 and DST Root CA X3, intermediate and root certificates will expire on Sep 29, 2021 and Sep 30, 2021 respectively. So since May 4, 2021, The newly issued certificates use a longer chain with cross-signed ISRG Root X1 as an intermediate certificate.

Unfortunately, due to the way certificate paths are built and verified, not all implementations of TLS can successfully verify the cross-sign. This is the case with OpenSSL 1.0.2. Hence, programs running on RHEL/CentOS 7 that use OpenSSL will likely fail to verify the new certificate chain or establish TLS connection. Upgrading to newer Openssl versions on such platforms is not straightforward.

There are a few options: either update the trust store (remove DST Root CA X3 root certificate - once it is removed, impact should be minimal) on the client side (or) change the certificate chain on the server side.

For Nginx

For Nginx there is only one parameter to specify the cert file. You should use the fullchain.pem provided by certbot to get it working correctly.

The right configuration in the server block for the given virtualhost would be as follow:

server {
  ...
  ssl_certificate /etc/letsencrypt/live/co2-avatar.com/fullchain.pem;  -> replaced cert.pem for fullchain.pem
  ssl_certificate_key /etc/letsencrypt/live/co2-avatar.com/privkey.pem;
}

References

• DST Root CA X3 Expiration (September 2021)
• Let’s Encrypt change affects OpenSSL 1.0.x and CentOS 7

Comments

[BairDev]

Thanks! I am already using the fullchain file, exactly like SSLCertificateFile /etc/letsencrypt/live/co2-avatar.com/fullchain.pem. So the intermediate cert is somehow not present, which might explain the fail of openssl verify -CAfile .../fullchain.pem .../cert.pem. How do you see this: Whats missing is the intermediate certificate? I guess that I need to fix the certificate(s). Any hint for that?

[Lutz Willek]

Answer your second question: The openssl s_client ... -showcerts command shows you all certificates sent by a server. In your case: The fact that only one certificate is returned shows that the intermediate(s) are missing. You can see it best if you compare it with the output in case you are using another domain. Example to compare with: echo | openssl s_client -connect belug.de:443 -servername belug.de -showcerts Check if all the expected certificates are included in your fullchain.pem file. (it is expected to see more than one certificate there)

[Lutz Willek]

To answer your first question: Given that the intermediate cert(s) present in file fullchain.pem and configured in your vhost, but not delivered by your server, strongly indicates that another vhost file is used. Check the other Apache configuration files - I would bet that the vhost configuration overlaps.

[BairDev]

Alright, now all SSLCertificateFile directives in my conf files point to fullchain.pem. But the second cert in exactly this file is not delivered. What is the SSLCertificateChainFile directive for (it is deprecated, right)? Some other configuration seems to block the fullchain.pem file. (yes, I have restarted apache / httpd)

[BairDev]

It turned out that the Apache version is too low / old (< 2.4.8). Unless we will update Apache, we need the full chain with SSLCertificateFile and SSLCertificateChainFile.