How do I prevent Bing from swamping my site with traffic irregularly?

Question

Bingbot will hit my site pretty hard for a couple of hours each day, and will be extremely light for the rest of the time.

I'd either like to smooth out its crawls, reduce its rate limit, or block it altogether. It doesn't really send through any real visitors.

Is there a way I can smooth its crawling, or rate limit it?

Solution

Their webmaster blog says that they support adding a crawl-delay parameter to your robots.txt file to throttle the bingbot.

User-agent: msnbot
Crawl-delay: 1

There's a bit more explanation in the webmaster FAQ PDF

These other links might be helpful as well:
http://www.bing.com/toolbox/webmasters
http://www.bing.com/community/webmaster/f/12252/t/651373.aspx

Comments

[knocte]

this has a bad side-effect: it will lower your pagerank