Error 429 with simple query on google with requests python

Question

I am trying to get the first non-ad result on a simple query on Google.

res = requests.get('https://www.google.com?q=' + query)

Assign any value to query and you will get an error. I have tried to add some headers, but nothing changes.

I have tried to add all other parameters that google typically associates to a query and again nothing changes.

No problems if you do the search with selenium.

The error code is 429, but this seems to be just a standard response for this query. It has nothing to do with my IP and I am not spamming Google, and this does not disappear after a while.

Do you know why this happens, and is there some header I can add, or any other solution to just see the results, as if you were searching that keyword on google?

Solution 1

429 Too Many Requests

The HTTP 429 Too Many Requests response status code indicates that the user has sent too many requests in a given amount of time ("rate limiting"). The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

When a server is under attack or just receiving a very large number of requests from a single party, responding to each with a 429 status code will consume resources. Therefore, servers are not required to use the 429 status code; when limiting resource usage, it may be more appropriate to just drop connections, or take other steps.

However, when I took you code and executed the same test, I got the perfect result as follows:

• Code Block:

  import requests
  
  query = "selenium"
  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}
  url = 'https://www.google.com?q=' + query
  res = requests.get(url, headers=headers)
  print(res)
  

• Console Output:

  <Response [200]>
  

You can find a relevant discussion in Failed to load resource: the server responded with a status of 429 (Too Many Requests) and 404 (Not Found) with ChromeDriver Chrome through Selenium

Solution 2

Since you are getting status code 429 which means you have sent too many requests in a given amount of time ("rate limiting"). Read in more detail here.

Add Headers in your request just like this:

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)\
            AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'}

So the final request will be:

url = 'https://www.google.com?q=' + query
res = requests.get(url, headers=headers)

Solution 3

I found reason why google simple query, rest-api request make 429 error.

user-agent header is one of reason, but I tried to insert user-agent header in request. but 429 error was made in response.

finally I found why, reason is cookies.

if you want access google page apis, first of all you have to get cookies from basic google urls like google.com, trend.google.com, YouTube.com. this basic site can be accessed by using any request method.

 googleTrendsUrl = 'https://google.com'
 response = requests.get(googleTrendsUrl)
 if response.status_code == 200:
    g_cookies = response.cookies.get_dict()

and this cookies insert into api request with user-agent

  headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)\
            AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'}
  url = 'https://www.google.com?q=' + query
  res = requests.get(url, headers=headers, cookies=g_cookies)

Comments

[Adrian Nicoli]

I had user-agent in my headers yesterday, and it didn't work, but now it is. I would not say that I am sending too many requests, and just requesting "google.com" would give me status 200 every time. Maybe something that is relevant to say is that I am using a VPN. But I am going to do more reading on the topic, thanks

[Sasha1296]

Adrian Nicoli, same thing for me. Did you ever figure it out? perhaps it's possible to wait a certain time before sending a new request, I'm not sure how long though