How do I force LWP to use Crypt::SSLeay for HTTPS requests?

Question

My symptom is that I cannot use a proxy with HTTPS requests with LWP. This seems to be a common problem, and the hints on Google and even here all suggest a work-around for setting the HTTPS_PROXY environment variable for use by Crypt::SSLeay.

My specific problem appears to be that LWP::Protocol::https is loading IO::Socket::SSL rather than Crypt::SSLeay. How can I force Crypt::SSLeay's use instead?

My code:

#!/usr/bin/perl

use strict;
use warnings;
$ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

And it's output, showing that Crypt::SSLeay is not being used:

Net/SSLeay.pm
IO/Socket/SSL.pm
/usr/lib/perl5/auto/Net/SSLeay/autosplit.ix
/usr/lib/perl5/auto/Net/SSLeay/set_proxy.al
/usr/lib/perl5/auto/Net/SSLeay/randomize.al

Simply adding an explicit use Crypt::SSLeay to my script has proven ineffective. It loads the module, but it continues to load IO::Socket::SSL, and use it for the HTTPS requests.

Solution 1

Try this:

use strict;
use warnings;

use Net::SSL (); # From Crypt-SSLeay
BEGIN {
  $Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
  $ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
}

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;

I don't have a suitable proxy, so I haven't tried it myself.

Solution 2

This is what I did to get LWP and SOAP::Lite to work with our proxy at GE. This was after much digging on CPAN, google etc. I finally figured it out after running the test script in the Crypt::SSLeay package called net_ssl_test and it was able to connect through the proxy. The key is forcing Crypt::SSLeay to use Net::SSL as was mentioned above. This is not documented very well on CPAN however.

use LWP::UserAgent;

# override HTTPS setting in LWP to work with a proxy
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY}               = 'http-proxy.ae.ge.com:80';
$ENV{HTTPS_PROXY_USERNAME}      = 'username';
$ENV{HTTPS_PROXY_PASSWORD}      = 'password';

$ua = new LWP::UserAgent;

# make a https request
my $req = HTTP::Request->new(GET => 'https://mail.google.com/');
my $res = $ua->request($req);
print $res->as_string;

Comments

[Flimzy]

Perfect, thanks! Doing this revealed the likely reason that LWP defaults to IO::Socket::SSL, as you touched on in your own question: Net::SSL from Crypt-SSLeay can't verify hostnames; either install IO::Socket::SSL or turn off verification by setting the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable to 0

[brian d foy]

LWP 6 changed it's policy on validating host names. If you can, use something other than Crypt::SSLeay, which doesn't have a dedicated developer (although Sinan has been helping it compile).

[Flimzy]

@briandfoy: If I can figure out how to use IO::Socket::SSL (or another option--are there other options?) with a proxy, I'll do that.

[AnBisw]

@cjm so in the code you are setting $ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128'; that's not a secured proxy. It should be $ENV{HTTPS_PROXY} = 'https://10.0.3.1:3128'; ?????? isn't that right?

[cjm]

@Annjawn, well that depends on whether your proxy listens via https. I just copied the proxy setting from the question.

[AnBisw]

OK, I thought $ENV{HTTPS_PROXY} is for https proxies and $ENV{HTTP_PROXY} is for usual http proxies.