Does googlebot keep sessions when crawling?

10,370

Solution 1

The answer to one of your question is: yes, you will have problems with Google bot.

Generally we've encountered two types of issues with google bot:

  1. it sometimes does not retain HTTP cookies between requests. Our application relies on custom cookies and the there were plenty of google bot requests caught to carry no cookies at all.

  2. it makes long breaks between consecutive requests. For example, it retrieves your page and asks for it's scripts later on.

Both will cause troubles with your session. First - you need a precise ASPNETSessionID cookie to be passed between requests. Googlebot will probably sometimes fail to do that. Second - if there's a long timespan between requests, your session is going to terminate even if the cookie is there.

Solution 2

Googlebot actively tries to avoid sessions and does not support cookies. From First date with the Googlebot: Headers and compression (March 2008)

I usually avoid cookies (so no "Cookie:" header) since I don't want the content affected too much by session-specific info. And, if a server uses a session id in a dynamic URL rather than a cookie, I can usually figure this out, so that I don't end up crawling your same page a million times with a million different session ids.

I imagine most regular search engine bots will be similar in this respect. Google is trying to build an index of unique URLs. The URL is the unique key that identifies a unique page of content. Cookies (and sessions) are not passed when a user clicks a link in the SERPS. Google is primarily indexing pages, not sites.

Solution 3

Generally the answer is no, however other crawlers (of which there are plenty) work other ways.

I should note that I have seen an instance of a google crawler for Adwords (not the normal googlebot) which DID present a session cookie.

Share:
10,370

Related videos on Youtube

MonsterMMORPG
Author by

MonsterMMORPG

Hello. I am the only owner and developer of web based online MMORPG game MonsterMMORPG. I am a computer engineer from Turkey and i am currently doing MA at computer engineering. I am specialized with C# & ASP.net. http://www.monstermmorpg.com/ MonsterMMORPG is a Free To Play Browser Based Online Monster MMORPG Game Better Than Online Pokemon Games You will love it's awesome Monsters We have many different unique features. So i suggest you to check them out. Our game is similiar with Pokemon games but it is unique. Like diablo and torch light. You should visit following sites related to us MonsterMMORPG Facebook Pokemon Games Lovers Facebook Application MonsterMMORPG Youtube Channel Monster Game Forum Canavar Oyunu Forum Pokemon Fakemon DeviantArt MonsterMMORPG Google Plus MonsterMMORPG Twitter MonsterMMORPG Review On Browsergamez MonsterMMORPG Review On mmohuts MonsterMMORPG Developer Blog At MMORPG.com MonsterMMORPG Review On onrpg MonsterMMORPG On GameSpot MonsterMMORPG Wiki MonsterMMORPG On 1UP MonsterMMORPG Digg MonsterMMORPG Official Google Plus Page

Updated on June 04, 2022

Comments

  • MonsterMMORPG
    MonsterMMORPG over 1 year

    When googlebot crawls pages does it have session? For example I am storing some variables on the session and using them in my site's pages. When googlebot crawls these pages will I still have the session-variables? In my global.asax I am storing some variables on the session at session start. Will I have any problem with Google bot?

  • MonsterMMORPG
    MonsterMMORPG about 12 years
    you mean for each page request ?
  • Icarus
    Icarus about 12 years
    @MonsterMMORPG My bet is that the Google Bot will use the query string to keep track of the same session while it's crawling your site; however, once it leaves your site and comes back, will very like do it under a different session. I have difficulties imagining a scenario where maintaining session info would be beneficial for the crawler, and in contrast, I can imagine the technical challenges that it would represent to try an maintain session info.
  • Buttle Butkus
    Buttle Butkus about 8 years
    @Icarus no, it would be idiotic for it to crawl a stateful version of your site when its whole purpose is to present stateless links for the world to see. If a bot loads up its cart with 7 wonderful widgets and then presents its current url as a search result for widgets, the search engine's users will just see an empty cart. Do you see?
  • Buttle Butkus
    Buttle Butkus about 8 years
    It's doubtful that any search engine spider would send cookies. If they do, then avoid those search engines because their results will be garbage.