HTTPS Proxy and LWP::UserAgent
Solution 1
I just uploaded the LWP::Protocol::connect module to CPAN. This module adds the missing HTTP/CONNECT method support to LWP.
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');
$ua->get('https://www.somesslsite.com');
With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.
Solution 2
Why would you want "Force use of Net::SSL". Try
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
# $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print $response->code ."\n";
An out put of 200
should mean that there were no errors.
A below sample code of mine works perfectly
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080'; #Valid HTTPS proxy taken from http://hidemyass.com/proxy-list/
$ENV{HTTPS_DEBUG} = 1;
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://www.nodeworks.com');
my $res = $ua->request($req);
print $res->code, "\n";
Output-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
With https://github.com/
the output is-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
So having said all this. Your code version (below) should work fine-
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080';
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; #works even with this
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://github.com/');
my $res = $ua->request($req);
print $res->code, "\n";
if ($res->is_success) {
print $res->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $res->status_line ."\n";
exit(1);
}
Solution 3
I pretty much ran into the same problem. Here are the things that fixed it for me:
- make sure you use Crypt::SSLeay 0.58 or lower. At least 0.64 blocked endlessly for me (as described at http://cpanforum.com/posts/13808 ), while 0.57/0.58 worked.
- make sure Net::HTTP has been updated to at least 6.03 (to avoid https://rt.cpan.org/Public/Bug/Display.html?id=72790 )
Solution 4
Instead of using Net::SSL which does not provide much host verification (and no SNI) you can use Net::SSLGlue::LWP. This monkey-patches LWP so that https_proxy can be used with the default SSL backend IO::Socket::SSL:
use Net::SSLGlue::LWP; # do this first
use LWP::Simple;
... continue with normal LWP stuff..
Ted B
Updated on July 30, 2020Comments
-
Ted B almost 4 years
I have read a number of threads on a number of sites and am still unable to make this work.
I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an HTTPS site (https://github.com in the example) via a WinGate proxy that I have running on a Windows VM. Note that my actual application is attaching to an SSL webservice that I have no control over.
From Firefox, pointed to the proxy everything is copacetic. The page loads successfully and I see the connections in the Proxy software Activity monitor. I'll be darned if I can make it work in Perl though. I've started with the code from this Stack Overflow Question : How do I force LWP to use Crypt::SSLeay for HTTPS requests? And added some debugging and additional output. Here's were I stand now:
#!/usr/bin/perl use strict; use warnings; use Net::SSL (); # From Crypt-SSLeay BEGIN { $Net::HTTPS::HTTPS_SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL $ENV{HTTPS_PROXY} = 'https://192.168.1.11:80'; # $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; $ENV{HTTPS_DEBUG} = 1; #Add debug output } use LWP::UserAgent; my $ua = LWP::UserAgent->new(); my $req = HTTP::Request->new('GET','https://github.com/'); my $response = $ua->request($req); print "--\n"; print "$_\n" for grep { $_ =~ /SSL/ } keys %INC; print "--\n"; if ($response->is_success) { print $response->decoded_content; # or whatever exit(0); } else { print "\nFail:\n"; print $response->status_line ."\n"; exit(1); }
Here's the output from this code:
-- Crypt/SSLeay.pm Crypt/SSLeay/X509.pm Net/SSL.pm -- Fail: 500 Can't connect to github.com:443 (Crypt-SSLeay can't verify hostnames)
If I then uncomment
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
, I do see a single connect to github.com:443 on the proxy and then nothing. (Note it works great from a web browser through the proxy). After much hanging I get the following output from the script:SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:before/connect initialization SSL_connect:SSLv3 write client hello A SSL_connect:failed in SSLv3 read server hello A SSL_connect:before/connect initialization SSL_connect:SSLv2 write client hello A SSL_connect:failed in SSLv2 read server hello A -- Crypt/SSLeay.pm Crypt/SSLeay/X509.pm Net/SSL.pm Crypt/SSLeay/CTX.pm Crypt/SSLeay/MainContext.pm -- Fail: 500 SSL negotiation failed:
If anyone can provide some direction here I would greatly appreciate it!