how to send a request to Google with curl

Question

I work on Linux and try to use curl to send requests to Google and save its reply as a html file.

When I use Google to search something, such as a string "abc", I find that the link of Google is: https://www.google.lu/#q=abc

So I try like this:

curl https://www.google.lu/#q=abc -o res.html

But the res.html is just the main page of Google, instead of the result of searching "abc".

How to do it?

Solution

Anything after the # is handled client side with JavaScript, which is why it doesn't work with curl.

You can instead use the traditional, non-AJAX interface on https://www.google.com/search?q=abc

It appears to block you unless you also spoof the user agent, so all in all:

curl \
  -A 'Mozilla/5.0 (MSIE; Windows 10)'  \
  -o res.html \
  "https://www.google.com/search?q=abc"