Why does accessing a SSL site with Mechanize on Windows fail, but on Mac work?

Question

This is the code I'm using to connect to the SSL site.

require 'mechanize'
a = Mechanize.new
page = a.get 'https://site.com'

I"m using using Ruby 1.9.3 and Mechanize 2.1pre1 + dependencies. On Mac the above code works and returns the page. On windows 7 running the same versions it gives me the following error:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3
read server certificate B: certificate verify failed

Reverting to Mechanize 2.0.1 seems to solve this problem, but I then get plagued with the too many connections reset by peer problem. Thus that is not a solution.

I've tried doing a.verify_mode = false, but that does not do anything. I have read that you can turn off SSL verification by using:

open(uri,:ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE)

How can I turn it off in Mechanize ? Why am I only getting this error on Windows ?

Solution 1

The version of OpenSSL (the library used to establish secure connections with Net::HTTPS) is not able to properly find the certificate chain in your computer.

To our bad, OpenSSL was never able to use the Windows installed cert storage to validate remote servers so is failing because of that.

From your example, you can do:

a.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE

To avoid the verification, however that is far from ideal (due clear security issues)

I recommend you download some cert bundles (like the ones from curl):

http://curl.haxx.se/ca

And modify your code to something like this:

require "rbconfig"
require "mechanize"

a = Mechanize.new

# conditionally set certificate under Windows
# http://blog.emptyway.com/2009/11/03/proper-way-to-detect-windows-platform-in-ruby/
if RbConfig::CONFIG["host_os"] =~ /mingw|mswin/
  # http://curl.haxx.se/ca
  ca_path = File.expand_path "~/Tools/bin/curl-ca-bundle.crt"

  a.agent.http.ca_file = ca_path
end

page = a.get "https://github.com/"

That seems to work, Ruby 1.9.3-p0 (i386-mingw32), Windows 7 x64 and mechanize 2.1.pre.1

Hope that helps.

Solution 2

Luis' answer looks fine but more generally:

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

Comments

[Luis Lavena]

You should avoid doing VERIFY_NONE, that defeats the purpose of using HTTPS/SSL in the first place if you're not verifying the other party. If you we're connecting through a transparent proxy or modified DNS that redirects you to a place that fake is the real one, you will never know...

[pguardiario]

Yes but you generally don't care about such things when scraping a webpage.

[Kassym Dorsel]

If you are testing SSL of a known host checking if it's valid is not necessary. As long as the encryption still holds ? Would you know how to add this conditionally if the platform is windows ?

[Luis Lavena]

@KassymDorsel updated the code to use platform conditional, hope that helps.

[RAJ]

I used a.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE for many projects but now when I am using same thing it doesn't effect. Still I am getting SSL error OpenSSL::SSL::SSLError Exception: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: (null) while in other projects it works fine... ruby 1.9.2 and rails 3.2.0

[RAJ]

Still getting same error OpenSSL::SSL::SSLError Exception: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: (null)

[Luis Lavena]

@RAJ... dunno what version of Mechanize are you trying but latest version has simplified the access to the certs. Please check Mechanize documentation about using certs.

[Prathan Thananart]

@RAJ... I got the exact same cryptic error read server hello A: (null under Ruby 1.9.3. Finally downgraded to 1.9.2-p290 and everything works fine. Very weird. Sorry to threadjack guys but this is the only Google result.

[RAJ]

Simple http request is working with use_ssl = true option. But not worked with Mechanize. So, I removed mechanize and used simple http requests

[Prathan Thananart]

@RAJ... Yeah, I just ran into this again with a different site. So I downgraded to Mechanize 2.0 and now it works with every site.

[RAJ]

@PrathanThananart It may help, I will check it later... because now I am not using Mechanize in my project... Thanks for follow up...

[Bryce Anderson]

This worked for me (the general approach, at least). I knew a validating root CA was at /opt/local/etc/openssl/cert.pem, so telling the mecha to use that file resolved the error securely. But can anyone tell me what cert bundle Mechanize uses by default? As best I can tell, it's not using the one that came with MacPorts openssl, the OSX keychain, or the bundle that comes with curl. The gem itself doesn't seem to include one.

[Hassan Akram]

For Mechanize v=2.7.x, self.m_agent.verify_mode = OpenSSL::SSL::VERIFY_NONE