NET::ERR_CERT_REVOKED in Chrome, when the certificate is not actually revoked

Question

I am looking for some help, trying to satisfy my curiosity by figuring out why Chrome 46.0.2490.80 won't let me access https://www.evernote.com, while Firefox works fine. Chrome was working fine too, until 2 days ago, but now it throws the NET::ERR_CERT_REVOKED error.

So I got curious - is the certificate actually revoked? Well let's check...

I opened the certificate dialog and exported the certificate (evernote.pem), and it's issuer chain (evernote-chain.pem):

Grab the OCSP Responder URI from the certificate:

$ openssl x509 -noout -ocsp_uri -in evernote.pem
http://ss.symcd.com 

Let's check the certificate status now:

$ openssl ocsp -no_nonce -issuer evernote-chain.pem -CAfile evernote-chain.pem -cert evernote.pem -url http://ss.symcd.com
Response verify OK
evernote.pem: good
        This Update: Dec 16 09:14:05 2015 GMT
        Next Update: Dec 23 09:14:05 2015 GMT

So, the certificate is not revoked, which is why Firefox works correctly. So what's going on with Chrome? Why does it think that this certificate is revoked?

I did notice another detail, which may or may not be important - I don't really understand it. The chain of certificates in Chrome is different from the chain obtained from Firefox, or from openssl. Chrome is seeing the following chain:

|- Class 3 Public Primary Certification Authority (Builtin Object Token, self-signed)
|---- VeriSign Class 3 Public Primary Certification Authority - G5 (35:97:31:87:F3:87:3A:07:32:7E:CE:58:0C:9B:7E:DA)
|------- Symantec Class 3 Secure Server CA - G4
|---------- www.evernote.com

Firefox and openssl see this instead:

|- VeriSign Class 3 Public Primary Certification Authority - G5 (18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A, self-signed)
|---- Symantec Class 3 Secure Server CA - G4
|------- www.evernote.com

I am not sure how to interpret this. It seems VeriSign Class 3 Public Primary CA is almost the same in Chrome, except instead of being self signed, it is replaced with something that looks exactly like it, but signed by this other "Builtin Object Token" in Chrome... What does this mean? Could it have anything to do with the problem I am experiencing?

UPDATE:

The first part of the question has been answered below. The reason why the site stopped working recently is b/c Google decided to distrust the “Class 3 Public Primary CA” root certificate, as explained here: https://googleonlinesecurity.blogspot.ca/2015/12/proactive-measures-in-digital.html

and here: https://code.google.com/p/chromium/issues/detail?id=570892

Since the decision was reversed for now, the problem can be fixed by getting the latest CRLSet in chrome://components/

However, the second part of the question remains:

Where is Chrome getting it's certificate chain from?

Firefox, openssl and https://www.digicert.com/help/, all show this same chain:

VeriSign Class 3 Public Primary Certification Authority - G5
18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A

Symantec Class 3 Secure Server CA - G4
51:3F:B9:74:38:70:B7:34:40:41:8D:30:93:06:99:FF

www.evernote.com
18:A9:E9:D2:F7:F4:D9:A1:40:23:36:D0:F0:6F:DC:91

Yet Chrome is using:

Class 3 Public Primary Certification Authority
70:BA:E4:1D:10:D9:29:34:B6:38:CA:7B:03:CC:BA:BF
- This is the no longer trusted Root CA

VeriSign Class 3 Public Primary Certification Authority - G5
35:97:31:87:F3:87:3A:07:32:7E:CE:58:0C:9B:7E:DA   <- WTF?!

Symantec Class 3 Secure Server CA - G4
51:3F:B9:74:38:70:B7:34:40:41:8D:30:93:06:99:FF

www.evernote.com
18:A9:E9:D2:F7:F4:D9:A1:40:23:36:D0:F0:6F:DC:91

The only explanation I can think of is that the "evil" version of the "VeriSign Class 3 Public Primary Certification Authority - G5" certificate is already in my certificate store somewhere. It has exactly the same CN and "Authority Key Identifier" as the "good" version, but it references a CA that is no longer trusted by Chrome. I have verified this assumption by direct comparison of the certificate in question between Chrome and Firefox. They are identical, and must generated using the same private key (or they wouldn't both be correctly verifying the signature on the Symantec cert), but one is self signed (the good one), and the other is not (the evil one).

So where is this certificate store/cache, which Chrome is using? Is it internal, or would it be system wide on Ubuntu? I am pretty sure that if I was to find and clear this cache, www.evernote.com would send me a full certificate chain during the next TLS handshake, and everything would right itself (this seems to support my theory: https://security.stackexchange.com/questions/37409/certificate-chain-checking).

But how do I blow away all of my cached certificates in Chrome?

Solution 1

Not sure what it all means, but the answer is there:

https://code.google.com/p/chromium/issues/detail?id=570892

and

https://googleonlinesecurity.blogspot.ca/2015/12/proactive-measures-in-digital.html

Google revoked a Symantec certificate from Google products, but they have suspended the revocation following the type of issues you're describing (which I also experienced). Quoting the Chromium ticket:

First, the good news is the change has been temporarily reverted, and you should find access restored. You can force an update by going to chrome://components and under CRLSet, clicking "Update". You should have version 2698 or later to fix this issue.

Solution 2

To answer the second part of the question, i.e. why Chrome finds a different trust path than the others.

The server is actually sending the following certificates to the client:

www.evernote.com
18:A9:E9:D2:F7:F4:D9:A1:40:23:36:D0:F0:6F:DC:91

Symantec Class 3 Secure Server CA - G4
51:3F:B9:74:38:70:B7:34:40:41:8D:30:93:06:99:FF

VeriSign Class 3 Public Primary Certification Authority - G5
25:0c:e8:e0:30:61:2e:9f:2b:89:f7:05:4d:7c:f8:fd

Note that the serial number on the last certificate is different to both serial numbers you have in your text, which means that there are several certificates with the same subject and public key out there which all can be used to built the trust chain.

Depending on the certificates you have installed in your system and the SSL stack used for validation different trust path are possible. For instance with openssl 1.0.1 on Ubuntu 14.04 it will also use the longer trust path you've seen with Chrome, i.e. the one which ends with the locally installed CA "Class 3 Public Primary Certification Authority". With OpenSSL 1.0.2 the handling of multiple trust path was changed and it will now prefer the shorter path which actually ignores the "VeriSign Class 3 Public Primary Certification Authority - G5" sent by the server and instead ends the trust chain in the locally installed version of a similar certificate (same public key and subject, different serial number).

The version of Chrome you've used had the specific Symantec certificate removed and now one of the possible trust path contains a revoked certificate, which means that the path cannot be trusted any longer. The bug is probably that Chrome then uses this result as the final result instead of looking for a different trust path which still would be valid.

Comments

[Val Blant]

Thank you, Yann. This answers the first part of the mystery very well. I have confirmed that the revoked Symantec certificate fingerprint is indeed the root CA of the certificate chain Chrome is using. The second part of the question remains though - Why is Chrome using a different chain from all other tools I try? I'll update my question with the new info.

[Dmitry]

CRLSet does not update anything. I use IIS sertivicate on localhost. And the sertificate is added in every possible place... Looks like Chrome bug is happaning for several years...