Ubuntu 11.10, using wget/curl fails with ssl

Question

On a completely new install of Ubuntu I'm getting the following errors when using wget:

wget https://test.sagepay.com

--2012-03-27 12:55:12--  https://test.sagepay.com/
Resolving test.sagepay.com... 195.170.169.8
Connecting to test.sagepay.com|195.170.169.8|:443... connected.
ERROR: cannot verify test.sagepay.com's certificate, issued by `/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA':
Unable to locally verify the issuer's authority.
To connect to test.sagepay.com insecurely, use `--no-check-certificate'.

I've tried installing ca-certificates and configuring the ca-certs and they appear to all be setup in /etc/ssl/certs.

The same issue exists for cURL:

curl https://test.sagepay.com

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Which leads me to believe it's something wrong with openssl server wide.

wget and curl both work correctly locally on OSX and I have confirmed with a few people that it's working on their servers so I suspect it's nothing to do with the server I'm attempting to connect to.

Any ideas or suggestions on things to try to narrow it down?

Thank you

Edit As requested verbose output from curl

curl -Iv https://test.sagepay.com
* About to connect() to test.sagepay.com port 443 (#0)
*   Trying 195.170.169.8... connected
* Connected to test.sagepay.com (195.170.169.8) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

Edit 2 Using the hash from your comment I see this:

ubuntu@srv-tf6sq:/etc/ssl/certs$ ls -al 7651b327.0
lrwxrwxrwx 1 root root 59 2012-03-27 12:48 7651b327.0 -> Verisign_Class_3_Public_Primary_Certification_Authority.pem
ubuntu@srv-tf6sq:/etc/ssl/certs$ ls -al Verisign_Class_3_Public_Primary_Certification_Authority.pem
lrwxrwxrwx 1 root root 94 2012-01-18 07:21 Verisign_Class_3_Public_Primary_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
ubuntu@srv-tf6sq:/etc/ssl/certs$ ls -al /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
-rw-r--r-- 1 root root 834 2011-09-28 14:53 /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
ubuntu@srv-tf6sq:/etc/ssl/certs$ more /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCEDyRMcsf9tAbDpq40ES/Er4wDQYJKoZIhvcNAQEFBQAwXzELMAkG
A1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz
cyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2
MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV
BAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmlt
YXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhE
BarsAx94f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/is
I19wKTakyYbnsZogy1Olhec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0G
CSqGSIb3DQEBBQUAA4GBABByUqkFFBkyCEHwxWsKzH4PIRnN5GfcX6kb5sroc50i
2JhucwNhkcV8sEVAbkSdjbCxlnRhLQ2pRdKkkirWmnWXbj9T/UWZYB2oK0z5XqcJ
2HUw19JlYD1n1khVdWk/kfVIC0dpImmClr7JyDiGSnoscxlIaU5rfGW/D/xwzoiQ
-----END CERTIFICATE-----

But doing the steps myself I end up with a different hash:

strace -o /tmp/foo.out curl -Iv https://test.sagepay.com

and

grep ssl /tmp/foo.out
open("/lib/x86_64-linux-gnu/libssl.so.1.0.0", O_RDONLY) = 3
stat("/etc/ssl/certs/415660c1.0", {st_mode=S_IFREG|0644, st_size=834, ...}) = 0
open("/etc/ssl/certs/415660c1.0", O_RDONLY) = 4
stat("/etc/ssl/certs/415660c1.1", 0x7fff7dab07b0) = -1 ENOENT (No such file or directory)

readlink -f /etc/ssl/certs/415660c1.0
/usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt

more /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCEDyRMcsf9tAbDpq40ES/Er4wDQYJKoZIhvcNAQEFBQAwXzELMAkG
A1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz
cyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2
MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV
BAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmlt
YXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhE
BarsAx94f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/is
I19wKTakyYbnsZogy1Olhec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0G
CSqGSIb3DQEBBQUAA4GBABByUqkFFBkyCEHwxWsKzH4PIRnN5GfcX6kb5sroc50i
2JhucwNhkcV8sEVAbkSdjbCxlnRhLQ2pRdKkkirWmnWXbj9T/UWZYB2oK0z5XqcJ
2HUw19JlYD1n1khVdWk/kfVIC0dpImmClr7JyDiGSnoscxlIaU5rfGW/D/xwzoiQ
-----END CERTIFICATE-----

Any other ideas? Thank you for the help so far :)

Edit: Answered below

Solution 1

So it turns out that installing the ca-certificates package didn't install the one that I needed. I found this post about certificates being presented out of order. This seems to be the case with my request to sagepay.

The solution ended up being to install another CA certificate from Verisign. I'm not sure why this fixes the issue with it being out of order but it does, but I suspect the out of order issue really isn't a problem at all and it was infact because I was missing a certificate all along. The additional certificate is available in that post but I didn't want to blindly trust it. I've looked at the list of CA certificates from cURL's site and it is listed there so I do trust it.

The certificate:

Verisign Class 3 Public Primary Certification Authority
=======================================================
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCEHC65B0Q2Sk0tjjKewPMur8wDQYJKoZIhvcNAQECBQAwXzELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQcmltYXJ5
IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2MDEyOTAwMDAwMFoXDTI4MDgwMTIzNTk1OVow
XzELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAz
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ2RHP7gJYHyX3KqhEBarsAx94
f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaOIG+YD/isI19wKTakyYbnsZogy1Ol
hec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0GCSqGSIb3DQEBAgUAA4GBALtMEivPLCYA
TxQT3ab7/AoRhIzzKBxnki98tsX63/Dolbwdj2wsqFHMc9ikwFPwTtYmwHYBV4GSXiHx0bH/59Ah
WM1pF+NEHJwZRDmJXNycAA9WjQKZ7aKQRUzkuxCkPfAyAw7xzvjoyVGM5mKf5p/AfbdynMk2Omuf
Tqj/ZA1k
-----END CERTIFICATE-----

I put this in a file in:

/usr/share/ca-certificates/curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

I then modified the /etc/ca-certificates.conf and added the following line at the end:

curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

After that I ran the command:

sudo update-ca-certificates

Looking into the /etc/ssl/certs directory I see it correctly linked:

ls -al | grep cURL
lrwxrwxrwx 1 root root     69 2012-03-27 16:03 415660c1.0 -> Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem
lrwxrwxrwx 1 root root     69 2012-03-27 16:03 7651b327.0 -> Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem
lrwxrwxrwx 1 root root    101 2012-03-27 16:03 Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.pem -> /usr/share/ca-certificates/curl/Verisign_Class_3_Public_Primary_Certification_Authority-from_cURL.crt

And everything works!

curl  -I https://test.sagepay.com
HTTP/1.1 200 OK...

Solution 2

Can you do a curl -Iv https://test.sagepay.com and update your question with those results?

What I see on a Ubuntu 10.04 box is:

$ curl -Iv https://test.sagepay.com
* About to connect() to test.sagepay.com port 443 (#0)
*   Trying 195.170.169.8... connected
* Connected to test.sagepay.com (195.170.169.8) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
*    subject: 1.3.6.1.4.1.311.60.2.1.3=GB; 2.5.4.15=Private Organization; serialNumber=01045967; C=GB; ST=TYNE AND WEAR; L=Newcastle Upon Tyne; O=Sage (UK) Limited; OU=Sage; OU=Terms of use at www.verisign.co.uk/rpa (c)05; OU=Authenticated by VeriSign; OU=Member, VeriS

and so on, so that looks OK.

Note the CApath is /etc/ssl/certs. Can you run sudo update-ca-certificates ? That should be in the ca-certificates package. If that package is not installed, try sudo apt-get install ca-certificates. If the ca-certificates package is not installed, then Ubuntu's list of CA certificates is not installed, and you will get validation errors.

Edit:

I see I skipped over the part where you say you've installed the ca-certificates package. In that case, we really need to see the verbose output from curl -Iv.

Edit 2:

OK, I'm running this command:

strace -o /tmp/foo.out curl -Iv https://test.sagepay.com

This will dump out strace to /tmp/foo.out. Looking at the strace file for any mentions of "ssl", I see:

$ grep ssl /tmp/foo.out 
open("/lib/libssl.so.0.9.8", O_RDONLY)  = 3
stat("/etc/ssl/certs/7651b327.0", {st_mode=S_IFREG|0644, st_size=834, ...}) = 0
open("/etc/ssl/certs/7651b327.0", O_RDONLY) = 4
stat("/etc/ssl/certs/7651b327.1", 0x7fffbef10f20) = -1 ENOENT (No such file or directory)

That /etc/ssl/certs/7651b327.0 certificate is what's being used to validate the test.sagepay.com one. Following that:

$ readlink -f /etc/ssl/certs/7651b327.0
/usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt

Does /etc/ssl/certs/7651b327.0 exist on your system? Does /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_Primary_Certification_Authority.crt exist?

Comments

[EEAA]

It would be best to remove the solution from your question and post as an answer.

[Greg Spiers]

@ErikA Didn't know that was the protocol, fixed. Thanks :)

[Ladadadada]

Although the error message says that you can use --no-check-certificate to skip the certificate check, cjc's suggestion is better than that because it's fixing the error rather than ignoring it.

[cjc]

That /etc/ssl/certs directory is not empty and the file /etc/ssl/certs/ca-certificates.crt exists, right? Generally, that directory should be full of symlinks to the actual cert files that live in /usr/share/ca-certificates/ and its subdirectories. That /usr/share/ca-certificates/ directory also exists and is readable by the user you're executing curl as?

[Greg Spiers]

The directory is not empty, lots of symlinks as you say to /usr/share/ca-certificates. Dir for /etc/ssl/certs and /usr/share/ca-certificates are both world readable. Even running sudo curl ... results in the same issue with the certificates.

[cjc]

@GregSpiers, I updated with more commands. Note that I'm doing this on Lucid -- I don't have a 11.10 box lying around -- but I can't imagine this being very different between Lucid and Oneiric.

[Greg Spiers]

@cjc Thanks, I've followed your example and updated my original question with the output. I agree, it shouldn't be that different between those two releases.

[cjc]

@GregSpiers, OK, that's completely bizarre. Both symlinks appear to be pointing to the same file /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_P‌​rimary_Certification‌​_Authority.crt, but, if you run the cert through openssl x509 -in /etc/ssl/certs/whatever -text -noout, I'm getting different results. Note that /etc/ssl/certs/415660c1.0 doesn't exist for me. You might do better with ubuntu.stackexchange.com as they should have a better idea of things that have changed.

[cjc]

@GregSpiers, specifically, you may want to present the two certificates in a question and ask why they're different, and why curl is choosing one over the other.

[Greg Spiers]

@cjc Thanks again for all your help! I've worked out the solution, see edit 3. I don't think I could have figured it out without your help with the hashes and which certificate is actually being used.

[noelbk]

This didn't exactly work for me, but this did on Ubuntu 14.04: sudo ln -s /usr/share/ca-certificates/mozilla/Verisign_Class_3_Public_P‌​rimary_Certification‌​_Authority.crt /etc/ssl/certs/415660c1.0

[noelbk]

Ah, found it. Digging around, I found that after an update, all certs in /etc/ca-certificates.conf were deselected. I removed the leading "!" on each line, ran update-ca-certificates and wget worked again