Proxy with urllib2

Question

I open urls with:

site = urllib2.urlopen('http://google.com')

And what I want to do is connect the same way with a proxy I got somewhere telling me:

site = urllib2.urlopen('http://google.com', proxies={'http':'127.0.0.1'})

but that didn't work either.

I know urllib2 has something like a proxy handler, but I can't recall that function.

Solution 1

proxy = urllib2.ProxyHandler({'http': '127.0.0.1'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')

Solution 2

You have to install a ProxyHandler

urllib2.install_opener(
    urllib2.build_opener(
        urllib2.ProxyHandler({'http': '127.0.0.1'})
    )
)
urllib2.urlopen('http://www.google.com')

Solution 3

You can set proxies using environment variables.

import os
os.environ['http_proxy'] = '127.0.0.1'
os.environ['https_proxy'] = '127.0.0.1'

urllib2 will add proxy handlers automatically this way. You need to set proxies for different protocols separately otherwise they will fail (in terms of not going through proxy), see below.

For example:

proxy = urllib2.ProxyHandler({'http': '127.0.0.1'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')
# next line will fail (will not go through the proxy) (https)
urllib2.urlopen('https://www.google.com')

Instead

proxy = urllib2.ProxyHandler({
    'http': '127.0.0.1',
    'https': '127.0.0.1'
})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
# this way both http and https requests go through the proxy
urllib2.urlopen('http://www.google.com')
urllib2.urlopen('https://www.google.com')

Solution 4

To use the default system proxies (e.g. from the http_support environment variable), the following works for the current request (without installing it into urllib2 globally):

url = 'http://www.example.com/'
proxy = urllib2.ProxyHandler()
opener = urllib2.build_opener(proxy)
in_ = opener.open(url)
in_.read()

Solution 5

One can also use requests if we would like to access a web page using proxies. Python 3 code:

>>> import requests
>>> url = 'http://www.google.com'
>>> proxy = '169.50.87.252:80'
>>> requests.get(url, proxies={"http":proxy})
<Response [200]>

More than one proxies can also be added.

>>> proxy1 = '169.50.87.252:80'
>>> proxy2 = '89.34.97.132:8080'
>>> requests.get(url, proxies={"http":proxy1,"http":proxy2})
<Response [200]>

View more solutions

Comments

[Chris Stryker]

I get File "D:/Desktop/Desktop/mygoogl", line 64, site = url.urlopen('google.com) File "C:\Python26\lib\urllib2.py", line 124, in urlopen return _opener.open(url, data, timeout) AttributeError: ProxyHandler instance has no attribute 'open'

[dcrosta]

I missed a call to urllib2.build_opener()

[satoru]

Hi, @ZelluX, I only want the proxies setting enabled on some function, does that mean I have to install and uninstall the opener for every invocation of the function?

[ZelluX]

@Satoru.Logic Maybe you can write a decorator to simplify the install/uninstall process?

[satoru]

Seems there's no uninstall method in urllib2, but we can make one-time proxy settings; instead of installing the opener, we create a request object, and use a opener to open it.

[ccpizza]

@Satoru.Logic: I think the traditional approach is to configure an environment variable like HTTP_PROXY and then check in your code if it is defined using os.environ["HTTP_PROXY"].

[J'e]

don't forget the port number eg 3128 proxy = urllib2.ProxyHandler({'http': '127.0.0.1:3128'})

[Sergey M]

@satoru, you can mimic uninstall through this

[Jonathan Benn]

Shouldn't you have used e.g. os.environ['http_proxy'] in your lower two sets of examples?

[MikeSchem]

how does this know what port to use??

[eckes]

For everybody using the solution above and wondering why the created ProxyHandler isn't used: I needed to use this solution for getting things working because I created an additional context for SSL verification: stackoverflow.com/a/24766345/520162

[Aman Singh]

Hi @WaqarDetho How will one know what proxy addresses to use? Is it just some random ip addresses?

[Waqar Detho]

Hi @AmanSingh I did this long time ago. But as far as I remember I find these proxy addresses from the internet. I manually injected them in the code.