Basic and Form Authentication with Mechanize (Ruby)

Question

I'm trying to log into a site on the company intranet which has a basic authentication popup dialog box and form based authentication. This is the code I'm using (which results in a 401 => Net::HTTPUnauthorized error):

require 'rubygems'
require 'mechanize'
require 'logger'

agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") }
agent.user_agent_alias = 'Windows Mozilla'
agent.basic_auth('username','password')
agent.get('http://example.com') do |page|
  puts page.title
end

This results in a 401 => Net::HTTPUnauthorized error. Below is mech.log info:

I, [2011-03-13T17:25:21.900631 #22128]  INFO -- : Net::HTTP::Get: /index.asp?LogIn=yes&action=7
D, [2011-03-13T17:25:21.901631 #22128] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2011-03-13T17:25:21.901631 #22128] DEBUG -- : request-header: accept => */*
D, [2011-03-13T17:25:21.901631 #22128] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6
D, [2011-03-13T17:25:21.902631 #22128] DEBUG -- : request-header: connection => keep-alive
D, [2011-03-13T17:25:21.902631 #22128] DEBUG -- : request-header: accept-encoding => gzip,identity
D, [2011-03-13T17:25:21.902631 #22128] DEBUG -- : request-header: host => example.com
D, [2011-03-13T17:25:21.902631 #22128] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2011-03-13T17:25:21.903631 #22128] DEBUG -- : request-header: keep-alive => 300
D, [2011-03-13T17:25:22.813722 #22128] DEBUG -- : Read 24 bytes
D, [2011-03-13T17:25:22.814722 #22128] DEBUG -- : response-header: content-type => text/html
D, [2011-03-13T17:25:22.815722 #22128] DEBUG -- : response-header: connection => close
D, [2011-03-13T17:25:22.815722 #22128] DEBUG -- : response-header: www-authenticate => Negotiate, NTLM, Basic realm="example.com"
D, [2011-03-13T17:25:22.816722 #22128] DEBUG -- : response-header: date => Sun, 13 Mar 2011 11:55:21 GMT
D, [2011-03-13T17:25:22.817722 #22128] DEBUG -- : response-header: server => Microsoft-IIS/5.0
D, [2011-03-13T17:25:22.817722 #22128] DEBUG -- : response-header: content-length => 24
I, [2011-03-13T17:25:22.819723 #22128]  INFO -- : status: 401

At this point I'm only trying to get past the first basic authentication. I did notice one thing that is that the server is IIS and in the Mechanize documentation there is a public function called gen_auth_headers(), however in the code of the gem I'm using this function doesn't exist, plus what he is doing in the code is beyond me.

TIA, Nabs

Solution 1

auth method will be removed in mechanize release 3, so it must be replaced by add_auth and provide the URI where those credentials apply.

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
agent.user_agent_alias = 'Windows Mozilla'
agent.add_auth('http://example.com', 'username', 'password')
agent.get('http://example.com') do |page|
  puts page.title
end

Solution 2

Use Mechanize#auth

require 'rubygems'
require 'mechanize'
require 'logger'

agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") }
agent.user_agent_alias = 'Windows Mozilla'
agent.auth('username', 'password')
agent.get('http://example.com') do |page|
  puts page.title
end