How to fix issues with LetsEncrypt certificate chains on Windows Server?

Question

I've been using LetsEncrypt to generate certificates for my sites on Windows 2012 R2 server. It worked great, until recently when I renewed the certificates.

LetsEncrypt made a recent change where they swapped the intermediate certificate with name "Let's Encrypt Authority X1" for one with name "Let's Encrypt Authority X3". The issue is, the authority key for the updated certificate remained the same.

https://community.letsencrypt.org/t/upcoming-intermediate-changes/

So, when I renewed by server certificates, they now were issues by "X3" authority, however since the key was the same, Windows cert store seems to build the certificate chain with the first result it finds (alphabetically?) which turns out to be the old "X1" cert.

This is where the problem arises. For some clients/browsers (like Chrome), this is fine, they only look at the key for the intermediate certificate. However, other clients are more strict and also check the name, and then fail (X1 instead of X3).

My first step to fixing this was to remove the X1 intermediate certificate, and make sure all my server certificates were updated to be issued by X3. Now things look correct, at least in the certificate store in Windows (the chain correctly shows Root Authority -> X3 -> server cert).

The problem I'm stuck on now, and can't seem to figure out, is why clients continue to display the wrong certificate chain (X1). That intermediate certificate doesn't even exist on my server anymore that I can see.

I've tried the usual rebooting server, and also stumbled upon this similar post, tried the steps there a few times without any luck -

https://serverfault.com/a/706278/182874

Any clue what I could be missing? There seems to be some problem with IIS caching certificate chains, since I've tried connecting on multiple clients/machines and all have the same problem. Just have no clue how to clear this "cert chain cache", or if it even exists.

Solution 1

I've just gone through this issue on my server. This was on 2008R2.

1. Ensure all certs are using the new intermediate certificate as we're going to remove the old one
2. Remove X1 intermediate cert from the local computer and all local user accounts
3. Remove the secure binding in IIS for each certificate
4. Re-add each of the secure bindings in IIS

There maybe some additional issues for 2012R2, there's a solution posted at https://community.letsencrypt.org/t/iis-8-5-building-incorrect-chain-with-lets-encrypt-authority-x3/13320/84 that quite a few people are reporting to work

Solution 2

The other answer didn't work for me (after the September 30th certificate expiry event). Windows automatically readded the certificate. I don't know where it fetched it from, but it's undeletable.

What worked for me was to go into the extended properties of the "DST Root CA X3" certificate and select "Disable all purposes for this certificate". After that, I just had to restart the IIS and it served the correct chain.

Note: Even though windows showed the correct chain in the cert-manager, IIS still served the invalid (and expired) old one...

Comments

[garethTheRed]

It could be your clients caching (although you've used multiple clients!). If you can, run openssl s_client <servername>:443 against your IIS server (without the < and > symbols) which will show you the Distinguished Names (and more) of the certificates being sent by your IIS server. You'll need to scroll up to the first few lines of the command output.

[David]

I've used multiple clients, as well as multiple machines (one of which has never hit the site before now). So, doesn't seem a client-side issue. When I run OpenSSL it shows that the intermediate certificate being presented is still X1.

[garethTheRed]

And who signed your end-entity certificate? Was it X1 or X3? The new Letsencrypt subordinate CA certificate (X3) has a new name so is a completely new certificate as far as chain-building is concerned - it shouldn't get confused with X1. That would simply break PKI :-)

[garethTheRed]

No - Windows, just like every other client, needs to Distinguished Name of the issuer (the CA) to be in the Issuer field of the certificate. End of, no ifs or buts, period. It also needs the public key of the issuer to be matched with the private key that signed the certificate. Change either the key-pair or the DN and it's a different certificate. Clients that uses Microsoft's CAPI (this includes Chrome on Windows) will use the certificate's AIA field to find the correct subordinate CA certificate, while Mozilla based products won't. This may be why you're seeing different results?

[garethTheRed]

You need to look at your IIS server and make sure that's configured to send the correct certificates to relying parties. IIS's end-entity certificate and the X3 certificate. Until that's resolved, you will get strange results.

[David]

I wasn't talking about Windows as a client though, was just talking purely about how it builds the cert chain (as a host). In the cert store, when building a chain, Windows seems to totally ignore "Subject -> CN" on the intermediate certificate, instead purely relying on the Key alone. This is how it originally built the incorrect chain with X1 instead of X3 (both keys were the same, only the name changed). Both Chrome and Firefox are showing the server reporting X1 as the intermediate cert. Chrome for whatever reason doesn't have any issue with it though (shows as completely valid).

[David]

Well, that's my whole issue with the server, is figuring out why IIS is still serving up the wrong intermediate certificate. Even after the cert store now shows correct chain. Sooooo, back to square one? :P

[garethTheRed]

Did you remove and reapply the certificate bindings in IIS? I still think that Chrome is using the AIA extension in the server certificate to download the correct (X3) subordinate certificate from the issuer. Firefox won't do that and therefore is forced to use the (incorrect) one supplied by IIS which causes it to complain.

[David]

Yeap, tried multiple times. Even tried doing these steps - serverfault.com/a/706278/182874 Well, when I view the certificate details in Chrome, it displays X1 for intermediate cert in the chain. So, to me that looks like IIS is serving up incorrect cert in both cases, but Chrome is just less strict about it (doesn't look at name, only key).

[David]

I actually had forgotten I posted this question. I did end up getting things working a while back with the solution by Knagis in that thread you linked, specifically this post - community.letsencrypt.org/t/…

[Ginji]

It's important for answers to be saved as it creates a nice repository of searchable solutions, which is why I added my answer after suffering the same issue. Thanks for coming back and accepting it @David

[Shadi Namrouti]

For me, it was exactly the intermediate X1 cert in the user account "not the local account". Thank you.

[K. Frank]

This didn't work for me. Windows somehow automatically re-adds the certificates...