Python TooManyRedirects: Exceeded 30 redirects

Question

I have tried the following code in Python

url="http://www.realtor.com/realestateandhomes-search/Pittsburgh_PA/type-single-family-home/price-na-30000/sby-1/"
r=requests.get(url)

but it throws the eror

  File "C:\Users\Dana\AppData\Local\Programs\Python\Python35-32\lib\site-package
s\requests\sessions.py", line 630, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\Dana\AppData\Local\Programs\Python\Python35-32\lib\site-package
s\requests\sessions.py", line 630, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Users\Dana\AppData\Local\Programs\Python\Python35-32\lib\site-package
s\requests\sessions.py", line 111, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects, respon
se=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

Any help would be greatly appreciated

Solution 1

It simply means that your request got a response which was a redirect (an information that the page you were trying to reach is now located at a new spot). The requests library understands this per default and does not return this result but tries another request for the new location. Which again returned a redirect, etc.

To avoid never coming out of the requests call, there is a limit implemented for the number of redirects allowed before the process is aborted.

I assume there is an error on the site you are trying to request something from, probably a circular redirect.

You can tweak the requests library to not follow the redirects but instead return them, then you will not get this error (but of course redirect responses):

response = requests.get(url, allow_redirects=False)

Solution 2

Occasionally, not often, this can happen if you do not include the headers the server is expecting. If you mimic the headers, payload, user-agent, etc. with the additional options available in requests.get() you'll be less likely to get this error.

Example:

import requests

headers = {
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
}

requests.get('http://www.realtor.com/realestateandhomes-search/Pittsburgh_PA/type-single-family-home/price-na-30000/sby-1', headers=headers)

Comments

[Andrei]

Could you explain / give an example how can i mimic the headers, payload etc.

[Andrei]

I have tried using this information and there is no error in this case, but the page i try to scrap does not load within the variable. Is there a method to avoid this error but also have the page loaded in python ?

[B.Adler]

Sure, example added.

[Mo Hajr]

this saved my day thanks, for those who wonder which headers to add just make the request using the browser or something like httpie, check the request headers, and copy them like the in the example above

[Ice Bear]

@Andrei 100% correct, I am having the same thing. No data to scrape

[Ice Bear]

doesn't work, I tried changing to around 100 still gets the error. What's the difference let's say if we set to 1,000? would it do anything?