How to hide pages from Google crawler?

Question

Possible Duplicate:
What are the most important things I need to do to encourage Google Sitelinks?

I'm currently working on a website and need to keep certain pages hidden from Google crawler. How to make it so that search engines see only what I want them to see in a directory?

Also, you know how Google results also give you shortcut links, Like 'Login', 'About' etc... how to put these links to search result?

Solution 1

If you'd like Google to not list certian URLs on your site, add them to robots.txt in the root of your site.

Example robots.txt:
User-agent: *
Disallow: /donotindex/

Note that this is not secure, you're just "asking" Google to not index/list them. Google will comply with this request. However, sophisticated users and some poorly-behaved crawlers will use or list the URLs regardless.

If you need to hide the content you'll need to put it behind a login page (which crawlers can't use) or change your content based on the client (browser's) user-agent. I believe the latter is specifically disallowed by Google and they may punish your site for doing go.

Solution 2

You can create a robots.txt that disallows those locations.
Beware that not all crawlers will respect robots.txt.

Also, robots.txt is the first place an attacker will look.
You should protext those pages with a secure authentication system.

Solution 3

Robots.txt.

Comments

[Admin]

The questions pertaining to blocking Google's crawler can be found in many of the questions tagged "robots.txt"