failed connect or "certificate verify failed" on LWP HTTPS GET

perl https lwp

23,001

$ openssl s_client -connect api.betfair.com:443 < /dev/null > api.betfair.com.pem
$ openssl x509 -in api.betfair.com.pem -issuer_hash
eb99629b

Well, whaddayasay, it's the same stupid intermediate certificate 0xeb99629b I've seen missing before with other people, see comment above for details and how to get it.

Out of curiosity, what version of OpenSSL and ca-certificates are you running? What's your system version/vendor?

23,001

Author by

bohica

In the Perl world I currently maintain DBD::ODBC, DBIx::Log4perl, DBIx::LogAny and contribute to DBI and DBD::Oracle. I have also written some commercial ODBC and XML drivers for Easysoft Limited. I mostly use Perl and pl/sql these days with some sprinkling of C (mostly Perl XS) but in my past I've used Fortran, Macro32, Assembler, C++ and even wrote some in kernel streams drivers in C for various unixes.

Updated on July 09, 2022

Comments

bohica almost 2 years

I posted this problem on Perl Monks yesterday but it worked for everyone who tried it (see http://www.perlmonks.org/?node_id=909968). However, I was using a different URL hoping to simplify the problem.

I'm attempting to connect to api.betfair.com via HTTPS and they have a valid certificate which I've verified in my browser. I am running ubuntu and have 2 versions of Perl. The system one 5.10.0 works and 5.14.0 installed via perlbrew fails. The code is:

use LWP::UserAgent; 
use strict;
use warnings;

#$ENV{HTTPS_CA_FILE} = "/usr/share/ca-certificates/cacert.org/cacert.org.crt";

my $ua  = LWP::UserAgent->new; 
my $req = HTTP::Request->new(GET => 'https://api.betfair.com');
my $res = $ua->request($req);

print $res->headers_as_string;
print $res->content;

Running this under the system Perl 5.10.0 it works fine and I get:

Date: Fri, 17 Jun 2011 08:33:04 GMT
Accept-Ranges: bytes
ETag: W/"0-1307353787000"
Content-Length: 0
Content-Type: text/html
Last-Modified: Mon, 06 Jun 2011 09:49:47 GMT
Client-Date: Fri, 17 Jun 2011 08:33:04 GMT
Client-Peer: 84.20.200.10:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
Client-SSL-Cert-Subject: /C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
Client-SSL-Cipher: RC4-MD5
Set-Cookie: NSC_mc-80-qvcbqj.efgbvmu=ffffffff09208c5545525d5f4f58455e445a4a4229a0;expires=Fri, 17-Jun-2011 20:33:05 GMT;path=/;httponly

Running it under Perl 5.14.0 I get: ~~Content-Type: text/plain Client-Date: Fri, 17 Jun 2011 08:34:30 GMT Client-Warning: Internal response Can't connect to api.betfair.com:443~~

If I uncomment the setting of HTTPS_CA_FILE and rerun in 5.14.0 I get:

Content-Type: text/plain
Client-Date: Fri, 17 Jun 2011 08:35:09 GMT
Client-Warning: Internal response
Can't connect to api.betfair.com:443 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/LWP/Protocol/http.pm line 51.

I have Mozilla::CA installed at version 20110409. Mozilla::CA::SSL_ca_file() returns "/home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Mozilla/CA/cacert.pem" and it exists and is readable by me. I am using LWP 6.02 in Perl 5.14.0 and 5.836 in Perl 5.10.0. I read setting HTTPS_DEBUG=1 should output some debug info but it only does this (for me) when using Perl 5.10.0 and not 5.14.0.

I'm not an SSL guru by any means but I tried some things I found and they just make me more confused:

openssl verify -verbose -CAfile /home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Mozilla/CA/cacert.pem < /dev/null
unable to load certificate
10888:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: TRUSTED CERTIFICATE

openssl s_client -CAfile /usr/local/share/perl/5.10.0/Mozilla/CA/cacert.pem -connect api.betfair.com:443 < /dev/null
CONNECTED(00000003)
depth=1 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network
---
Server certificate
-----BEGIN CERTIFICATE-----
certificate snipped
sg==
-----END CERTIFICATE-----
subject=/C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
---
No client certificate CA names sent
---
SSL handshake has read 3068 bytes and written 303 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID: 81802384A47AF45D2D809A2D10041A4E0B4B4DD821507569216A199ED467B207
    Session-ID-ctx: 
    Master-Key: 50DEC11CD2FA57E9BFA95B0156905D2717A79F333A2028FCCCB0F1C32A6B35202A958CEF24D3D2332A00CDCD158B40FB
    Key-Arg   : None
    Start Time: 1308304989
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
DONE

UPDATE: I thought it was because I had PERL_UNICODE=SAL set but unsetting this does not fix the problem.

UPDATE: versions: Linux ubuntu 10.10 codename maverick openssl 0.9.80 (I believe up to date on my ubuntu distribution

bohica almost 13 years

updated op with openssl version and system. Not sure how to find ca-certificates version. Had the same problem with facebook martin-evans.me.uk/node/95 about a month ago but solved it differently. Thanks daxim - I knew you'd be the one to diagnose this - almost asked you directly.
bohica almost 13 years

openssl s_client -CAfile cert.pm -connect api.betfair.com:443 < /dev/null now works. not got perl working though - still trying.
bohica almost 13 years

HTTPS_CA_FILE=cert.pm perl ca2.pl where ca2.pl is the perl in op fails in same way.
bohica almost 13 years

Download cert you pointed at. sudo cp cert.pm /etc/ssl/certs/Verisign_Class_3_Secure_Server_CA_G2.pem; sudo c_rehash; perl ca2.pl still fails - same problem. symb link in /etc/ssl/certs is 4d241d64.0 -> Verisign_Class_3_Secure_Server_CA_G2.pem
daxim almost 13 years

I did not manage to reproduce your problem in a fresh Ubuntu 10.10 VM, sorry. -- dpkg --status openssl # Version=0.9.8o-1ubuntu4.4 dpkg --status ca-certificates Version=20090814 -- Since you made it work with s_client, you can now simply point LWP's SSL stack to the system certs dir: HTTPS_CA_DIR=/etc/ssl/certs perl...
bohica almost 13 years

Pointing a system certificates works - thanks and upvoted. This is a really annoying issue though. I wish I can decide where the issue might be so I could report it.
daxim almost 13 years

The problem is many things at once (if nothing else, Perl's culture of wanting to please everyone and PKI through CAs being a total crock) and it can't be solved with a simple report. Visit YAPC::EU, I'll submit a talk.