QT ssl error on hotmail.com: The issuer certificate of a locally looked up certificate could not be found

Question

When I use QtWebkit (which uses QSslSocket internally) to connect to www.hotmail.com, I got the following ssl errors after logging in:

1. The issuer certificate of a locally looked up certificate could not be found
2. The root CA certificate is not trusted for this purpose
3. No certifiates could be verified.

I could get the same errors at mail.yahoo.com before logging in. I have found that secure connection to these two servers triggers the ssl errors:

1. https://gfx8.hotmail.com
2. https://csc.beap.bc.yahoo.com

There is a small ssl demo program included with QT called securesocketclient. I can produce the same errors if I use this little client to connect to port 443 of the above two servers. I am 99% certain this is a bug with QT since we have produced this issue on many development systems. It affects linux, mac and windows QT. It affects QT 4.7.4 all the way to QT 4.8.4 (didn't try earlier versions). The bug is reproduced with openssl 0.9.8.

Some people might falsely claim that the bug is fixed by using openssl 1.0.0 with precompiled QT binary. That's wrong since the precompiled QT binary is compiled with openssl 0.9.8 header files. Openssl 0.9.8 is not binary compatible with openssl 1.0.0. Some of the structure memebers will be misinterpreted if you compile QT with one version of openssl but use another version of openssl binary. I did a lot of step-and-trace into QT and openssl source code to arrive at this conclusion.

Since hotmail and yahoo email are extremely popular websites, I'd consider this bug is a serious bug in terms of security for QT. I already reported it to QT bug system (https://bugreports.qt.io/browse/QTBUG-23625) and I doubt when digia will ever get their hands on it based on my past experience.

Please feel free to share your thoughts on how to fix this bug. I do know how to call ignoreSslErrors and pretend there is no problem. But that's not how openssl is intended to be used.

---

Some updates from further investigation on this issue.

The certificate path for https://gfx8.hotmail.com is:

• Root CA: Versign Class 3 Public Primary Certification Authority - G5
• Intermediate CA: Versign Class 3 Extended Validation SSL SGC CA
• Server Certificate: gfx-ecn.hotmail.com

The root CA does sit in the Windows certificate store and is loaded properly at startup. I guess perhaps somehow QT doesn't get the intermediate certificate. I run a test with the openssl command line program as shown below:

openssl s_client -showcerts -connect gfx8.hotmail.com:443

The output is here:

CONNECTED(000001AC) depth=0 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Washington, bu sinessCategory = Private Organization, serialNumber = 600413485, C = US, postalC ode = 98052, ST = Washington, L = Redmond, street = 1 Microsoft Way, O = Microso ft Corporation, OU = Windows Azure CDN, CN = gfx-ecn.hotmail.com

verify error:num=20:unable to get local issuer certificate verify return:1

depth=0 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Washington, bu sinessCategory = Private Organization, serialNumber = 600413485, C = US, postalC ode = 98052, ST = Washington, L = Redmond, street = 1 Microsoft Way, O = Microso ft Corporation, OU = Windows Azure CDN, CN = gfx-ecn.hotmail.com

verify error:num=27:certificate not trusted verify return:1

depth=0 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Washington, bu sinessCategory = Private Organization, serialNumber = 600413485, C = US, postalC ode = 98052, ST = Washington, L = Redmond, street = 1 Microsoft Way, O = Microso ft Corporation, OU = Windows Azure CDN, CN = gfx-ecn.hotmail.com

verify error:num=21:unable to verify the first certificate verify return:1

Certificate chain 0 s:/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Washington/businessCategory=Private Organization/serialNumber=600413485/C=US/postalCode=98052/ST=Washington/L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/OU=Windows Azure CDN/CN=gfx-ecn.hotmail.com i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA

-----BEGIN CERTIFICATE-----

MIIG0DCCBbigAwIBAgIQfRbsuTLd2GrmU38TPnVOCTANBgkqhkiG9w0BAQUFADCB vjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNjE4MDYGA1UEAxMv VmVyaVNpZ24gQ2xhc3MgMyBFeHRlbmRlZCBWYWxpZGF0aW9uIFNTTCBTR0MgQ0Ew HhcNMTIxMTA3MDAwMDAwWhcNMTQxMTA4MjM1OTU5WjCCAR0xEzARBgsrBgEEAYI3 PAIBAxMCVVMxGzAZBgsrBgEEAYI3PAIBAhMKV2FzaGluZ3RvbjEdMBsGA1UEDxMU UHJpdmF0ZSBPcmdhbml6YXRpb24xEjAQBgNVBAUTCTYwMDQxMzQ4NTELMAkGA1UE BhMCVVMxDjAMBgNVBBEUBTk4MDUyMRMwEQYDVQQIEwpXYXNoaW5ndG9uMRAwDgYD VQQHFAdSZWRtb25kMRgwFgYDVQQJFA8xIE1pY3Jvc29mdCBXYXkxHjAcBgNVBAoU FU1pY3Jvc29mdCBDb3Jwb3JhdGlvbjEaMBgGA1UECxQRV2luZG93cyBBenVyZSBD RE4xHDAaBgNVBAMUE2dmeC1lY24uaG90bWFpbC5jb20wggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQC+jRmBRv2iw1N2LirFdhgqmZ3G+BBc8gAn50O6TT1u zNqrjicf3KJ+BDHSGcnkysvWovwnUhDhMzAWf521iYi2lFZqC3txewGvjrKM0Gqz DhHrF+bzvNjyrIION89354cFxU1eK2okegYHkWIuyPHyCN6PFGK52OlkBixb34xv WvAZfjSu/hr+f+lkedFZvJdd6KS4e8N/tGJ/dndfmReaKSiNFWBFbwhkndLdXU3p ZnVLCysETMuUoIsDIPcgDfji1XkoKLsri9WijVhjNH1MFf/t6/g4PpWqZGl4si3t yQ0rdefDGfgX8lvq63aXnaap4SbjTYLviFRle/PMkXV7AgMBAAGjggJmMIICYjCB 0AYDVR0RBIHIMIHFghBnZngxLmhvdG1haWwuY29tghBnZngyLmhvdG1haWwuY29t ghBnZngzLmhvdG1haWwuY29tghBnZng0LmhvdG1haWwuY29tghBnZng1LmhvdG1h aWwuY29tghBnZng2LmhvdG1haWwuY29tghBnZng3LmhvdG1haWwuY29tghBnZng4 LmhvdG1haWwuY29tghRncmFwaGljcy5ob3RtYWlsLmNvbYIIYS5nZngubXOCE2dm eC1lY24uaG90bWFpbC5jb20wCQYDVR0TBAIwADAdBgNVHQ4EFgQUH0b2ApITW9WB /LA+oaz+2ZnW/dwwDgYDVR0PAQH/BAQDAgWgMD4GA1UdHwQ3MDUwM6AxoC+GLWh0 dHA6Ly9FVkludGwtY3JsLnZlcmlzaWduLmNvbS9FVkludGwyMDA2LmNybDBEBgNV HSAEPTA7MDkGC2CGSAGG+EUBBxcGMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3 LnZlcmlzaWduLmNvbS9jcHMwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMC BglghkgBhvhCBAEGCisGAQQBgjcKAwMwHwYDVR0jBBgwFoAUTkPIHXbvN1N6T/JY b5TzOOLVvd8wdgYIKwYBBQUHAQEEajBoMCsGCCsGAQUFBzABhh9odHRwOi8vRVZJ bnRsLW9jc3AudmVyaXNpZ24uY29tMDkGCCsGAQUFBzAChi1odHRwOi8vRVZJbnRs LWFpYS52ZXJpc2lnbi5jb20vRVZJbnRsMjAwNi5jZXIwDQYJKoZIhvcNAQEFBQAD ggEBAFdVsrzkxJ6aRnaGIO1hbCDlekEMCT6OTlZXckzZeaIrNfSFLXHe89pWkRr1 AKz43nnM0pLxVuEHRE9pMZH6Om7SjqU5BR1qd6xp+ZJhuJA2I2319PCSbKCpv67X 82J8/JKjH8e4fpOzb70dKUlNNr7x0aIMYuCq6unXXZQ5u83Uny42jcIQWLOlZRKC dYSqW3JalTYVZNvdEoQVuUEJJLcY1qMVJ9NFtdnrzrmcpK+52+nZQXbCkM7W8Vl1 WM/dbOnqsu0+SIPZ4Q2wIAnT1azmBvxZ2ULvzW98HIAn4/RdPuimnox8T9R2yrv1 xd5P6oAZWvmnX6e461m6HohhhDw=

-----END CERTIFICATE-----

Server certificate subject=/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Washington/businessCategory=Private Organization/serialNumber=600413485/C=US/postalCode=98052/ST=Washington/L=Redmond/street=1 Microsoft Way/O=Microsoft Corporation/OU=Windows Azure CDN/CN=gfx-ecn.hotmail.com issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA

No client certificate CA names sent

SSL handshake has read 1933 bytes and written 480 bytes

New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : SSLv3 Cipher : AES256-SHA Session-ID: 3CC559C15AF17B09346C371A1CB292DF77C272A37FDC4DF69EEE0EE9CC067B5C Session-ID-ctx: Master-Key: F626E23FDCC89B1329FD4F5D1ED5A940F0CB14A1C377BFDB6ABA6238B91F9C11390EC16FD117C090B3171FBEE762B792 Key-Arg : None PSK identity: None PSK identity hint: None Start Time: 1355423684 Timeout : 7200 (sec) Verify return code: 21 (unable to verify the first certificate)

From the output of openssl, we can see that the openssl program produces similar errors (which makes it look more like an openssl bug?). We also notice that the hotmail server only returns the server certificate but does not return the intermediate certificate. That's probably what caused the error. I tried the same ssl command with other https servers. They also return both server certificates and intermediate certificates. Some of the banking sites I've tested even returned the entire certificate chain all the way to the root CA. So, the question is, if the intermediate certificate is not returned, how and where am I supposed to get it from? How does other browsers like IE and chrome handle this?

---

More findings here.

Actually, both of the two servers I mentioned are used to download in-page components for hotmail and yahoo mail. The yahoo mail main login page itself does return the full certificate chain. When other browsers access the main page, it will store the intermediate certificates locally for later use. However, QTWebkit doesn't cache the intermediate certificates automatically, therefore causing the problem. If I access the server https://gfx8.hotmail.com directly with a fresh copy of firefox, I get the same error. But if I access https:/www.hotmail.com first and then log into hotmail, there is no error although some components are downloaded from the gfx8 server after login. I found QTWebkit neither loads the intermediate certificates from system certificate store, nor does it add intermediate certificates to the system certificate store. It doesn't cache intermediate certificates internally in memory either. That will cause problem for some servers which doesn't return the complete certificate chain.

I did try saving the intermediate certificates to PEM file and load it at startup of my application. It works fine on Windows and Linux. But somehow, it has no effect on Mac. Don't know why.

Solution

In order to support secure connections in your application, without ignorig ssl errors, you must have the root certificates of all the domains that the site you're loading is accessing, so for example if it accesses a CDN(chech in the net panel of your browser), get that server's certificates also (you can download them from chrome for example, by viewing the current site's certificates, and get the whole chain, but separately download them). After you stored all these certificates in one separate folder, add this code somewhere after the initialization of your application, it will load all of them at the beginning, so QT can access them:

import os

from PySide.QtNetwork import QSsl, QSslConfiguration, QSslCertificate
from PySide.QtCore import QFile, QIODevice


def load_certs(cert_path):
    # cert_path is a string "/path/to/cert/files"
    ssl_config = QSslConfiguration.defaultConfiguration()
    ssl_config.setProtocol(QSsl.SecureProtocols)

    certs = ssl_config.caCertificates()

    for cert_filename in os.listdir(cert_path):
        if os.path.splitext(cert_filename)[1] in ('.cer', '.crt', '.pem'):
            cert_filepath = os.path.join(cert_path, cert_filename)
            cert_file = QFile(cert_filepath)
            cert_file.open(QIODevice.ReadOnly)
            cert = QSslCertificate(cert_file)
            certs.append(cert)

    ssl_config.setCaCertificates(certs)
    QSslConfiguration.setDefaultConfiguration(ssl_config)

Now when a site requires certificate validation, it won't fail. However openssl has indeed problems with it's TLS implementation on Debian systems, and in those cases you will experience problems such as pages never load, or similar errors when you don't have valid certificates. In those cases you have to force using SSLv3 for those pages, to get them work. You can achieve that by binding a custom virtual function to QNetworkAccessManager's createRequest, and overriding the ssl protocol for each request. That way you don't have to rely on older ssl version for all sites(because you could set the default ssl protocol globally too, but that's not the best idea), only for those which are problematic:

class Browser(object):

    def __init__(self):
        self.network_manager = QNetworkAccessManager()
        self.network_manager.createRequest = self._create_request
        self.web_page = QWebPage()
        self.web_page.setNetworkAccessManager(self.network_manager)
        self.web_view = QWebView()
        self.web_view.setPage(self.web_page)
        self._override_ssl = None

    def _create_request(self, operation, request, data):
        ssl_protocols = {'sslv2': QSsl.SslV2,
                         'sslv3': QSsl.SslV3,
                         'tlsv1': QSsl.TlsV1,
                         'tlsv1sslv3': QSsl.TlsV1SslV3,
                         'unknownprotocol': QSsl.UnknownProtocol,
                         'secureprotocols': QSsl.SecureProtocols}

        if self._override_ssl is not None:
            ssl_config = QSslConfiguration.defaultConfiguration()
            ssl_config.setProtocol(ssl_protocols[self._override_ssl])
            request.setSslConfiguration(ssl_config)

        reply = QNetworkAccessManager.createRequest(self.network_manager,
                                                    operation,
                                                    request,
                                                    data)

        return reply

    def override_ssl(self, protocol_id):
        # protocol id is a string like 'sslv3' or 'tlsv1'
        self._override_ssl = protocol_id

Ok I typed most of the code from memory, please report if something doesn't work.