Cannot connect to proxy error on requests.get() or requests.post() in python

Question

I have two URLs to fetch data from. Using my code, the first URL is working, whereas the second URL is giving ProxyError.

I am using requests library in Python 3 and tried searching the problem in Google and here, but with no success.

My code snippet is:

    import requests

    proxies = {
      'http': 'http://user:[email protected]:xxxx',
      'https': 'http://user:[email protected]:xxxx',
    }

    url1 = 'https://en.oxforddictionaries.com/definition/act'
    url2 = 'https://dictionary.cambridge.org/dictionary/english/act'

    r1 = requests.get(url1, proxies=proxies)
    r2 = requests.get(url2, proxies=proxies)

url1 works fine, but url2 gives following error:

    ProxyError: HTTPSConnectionPool(host='dictionary.cambridge.org', port=443): Max retries exceeded with url: /dictionary/english/act (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))

Same happens on using request.post()

1. Please explain me why this is happening, and is there any difference between the handshaking of both the URLs?

2. urllib.request.urlopen is working fine, so I am explicity looking for answers using requests

Solution

I was able to illicit a valid response for url2 when using headers keyword argument with User-Agent string set to Chrome.

r2 = requests.get(url2, proxies=proxies, headers={'User-Agent': 'Chrome'})

To answer your first question, possible reason for this happening is related to server-side settings. It might be configured not to accept requests originating from unknown agents or requests with a missing User-Agent header.

Comments

[thepunitsingh]

Worked like charm. That means that the website is checking for the agent which is making a request, and will response only to a browser. Any link where I can study this behaviour?

[Phoenix]

Yes. That seems to be the case. Thanks for accepting my answer. :)

[Phoenix]

apache server specific example: serverfault.com/questions/251988/…

[Suraj Rao]

While this code may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply.