Python urllib over TOR?

Question

Sample code:

#!/usr/bin/python
import socks
import socket
import urllib2

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 9050, True)
socket.socket = socks.socksocket

print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()

TOR is running a SOCKS proxy on port 9050 (its default). The request goes through TOR, surfacing at an IP address other than my own. However, TOR console gives the warning:

"Feb 28 22:44:26.233 [warn] Your application (using socks4 to port 80) is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information. Consider using Socks4A (e.g. via privoxy or socat) instead. For more information, please see https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS."

i.e. DNS lookups aren't going through the proxy. But that's what the 4th parameter to setdefaultproxy is supposed to do, right?

From http://socksipy.sourceforge.net/readme.txt:

setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])

rdns - This is a boolean flag than modifies the behavior regarding DNS resolving. If it is set to True, DNS resolving will be preformed remotely, on the server.

Same effect with both PROXY_TYPE_SOCKS4 and PROXY_TYPE_SOCKS5 selected.

It can't be a local DNS cache (if urllib2 even supports that) because it happens when I change the URL to a domain that this computer has never visited before.

Solution 1

The problem is that httplib.HTTPConnection uses the socket module's create_connection helper function which does the DNS request via the usual getaddrinfo method before connecting the socket.

The solution is to make your own create_connection function and monkey-patch it into the socket module before importing urllib2, just like we do with the socket class.

import socks
import socket
def create_connection(address, timeout=None, source_address=None):
    sock = socks.socksocket()
    sock.connect(address)
    return sock

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)

# patch the socket module
socket.socket = socks.socksocket
socket.create_connection = create_connection

import urllib2

# Now you can go ahead and scrape those shady darknet .onion sites

Solution 2

The problem is that you are importing urllib2 before you set up the socks connection.

Try this instead:

import socks
import socket

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket

import urllib2
print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()

Manual request example:

import socks                                                         
import urlparse                                                      

SOCKS_HOST = 'localhost'                                             
SOCKS_PORT = 9050                                                    
SOCKS_TYPE = socks.PROXY_TYPE_SOCKS5                                 

url = 'http://www.whatismyip.com/automation/n09230945.asp'           
parsed = urlparse.urlparse(url)                                      


socket = socks.socksocket()                                          
socket.setproxy(SOCKS_TYPE, SOCKS_HOST, SOCKS_PORT)                  
socket.connect((parsed.netloc, 80))                                  
socket.send('''GET %(uri)s HTTP/1.1                                  
host: %(host)s                                                       
connection: close                                                    

''' % dict(                                                          
    uri=parsed.path,                                                 
    host=parsed.netloc,                                              
))                                                                   

print socket.recv(1024)                                              
socket.close()

Solution 3

I've published an article with complete source code showing how to use urllib2 + SOCKS + Tor on http://blog.databigbang.com/distributed-scraping-with-multiple-tor-circuits/

Hope it solves your issues.

Comments

[OJW]

Actually that was the code I originally had, and doesn't seem to work any differently

[Wolph]

@OJW: are you 100% sure that you don't import urllib2 somewhere else before that? The above code works perfectly for me if I simply paste it in some file.

[OJW]

By "it works", do you mean that it downloads the page via the proxy (that bit works for me too), or are you also watching the TOR STDOUT to confirm that DNS requests are going via TOR and not direct?

[Wolph]

@OJW: I didn't use TOR for my test but simply an ssh connection while monitoring the connection for DNS requests with Wireshark. Although there is a chance that I didn't see the DNS request because of the caching DNS server that I run locally.

[Wolph]

@OJW: here's a recipe for a tor dns proxy, just in case you can't get it working: koders.com/python/…

[OJW]

Just tried it with Wireshark here too, and I can see the DNS request going out in cleartext to default gateway, before a load of SSL communications between TOR and some random host.

[Wolph]

@OJW: indeed... after clearing my dns cache I saw the DNS request too :( The socksipy docs seem to be faulty. The problem seems to be in urllib2, I don't get any requests when manually passing the http headers to the socket.

[OJW]

Raised as bugs.python.org/issue11375

[OJW]

Python bug 11375 was closed as "invalid", so looks like this behaviour will never be fixed

[Mastergalen]

NOTE: The new SOCKS port seems to be 9150. I literally spent an hour, disabling the firewall etc. trying to figure out why it didn't connect...

[Mastergalen]

NOTE: The new SOCKS port seems to be 9150. I literally spent an hour, disabling the firewall etc. trying to figure out why it didn't connect...

[hoju]

I needed to remove the monkey patching to get it working in 2.7

[Aleksandr]

@Mastergalen, not really. It could be any port if you change the default settings in your TOR config. The default is SocksPort 9150 and ControlPort 9151.