What happens if I try to set a cookie on a bot?

9,730

Most bots don't accept cookies (including Googlebot), however, some bots do.

You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. However, this information is not used in the search results.

When Googlebot visits your site it's like they have never visited before, which goes for any real users who have disabled their cookies.

Further reading: Does googlebot keep session when crawling?


One way to handle the situation of generating a unique visitor ID (guid) is... if the cookie doesn't exist (either a new user or the cookie hasn't been set for whatever reason) then attempt to set a cookie with your real guid, but use a hash of the user agent and IP address as the guid for the current request (which is distinguishable from the real guid). It's not perfect (it's not entirely unique), but it at least allows you to more easily track the bots and importantly avoids counting 1000's of unreal visitors (if you are relying solely on a cookie to count visitors).

Share:
9,730

Related videos on Youtube

Cyberherbalist
Author by

Cyberherbalist

Updated on September 18, 2022

Comments

  • Cyberherbalist
    Cyberherbalist about 1 year

    I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (including his user-agent, referer (if any), and so on, in my database. I plan to check each user to see if they have the cookie I've set, and then create a visit record each subsequent time they come by, to include if they click any links on my site.

    The site is very very new (only about a week old as of today), but I've been tracking my visitors in a text log and I see that I can identify visiting bots by their user-agents (the Google and Bing bots/crawlers have come by, along with a couple of others).

    It occurs to me that they probably won't accept cookies. Does anyone know this for sure? Or do they also collect cookies in addition to site information?

    • Admin
      Admin over 9 years
      Are you trying to restrict your site crawling by using cookies? Bots will always crawl your site regardless of do they accept cookies or not.
    • Admin
      Admin over 9 years
      No, I am not trying to restrict anything. Cookies are just to detect who has been to the site previously and to try to track what they do on the site. I am just curious if anyone knows if crawlers or bots are known to accept cookies, so I can perhaps plan for that.
    • Admin
      Admin over 9 years
      I don't know sure answer, but I'd not rely on accepting cookies by bots.