How to fix robots.txt file error in my GWT?

Question

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file.

My blog’s robots.txt file is:

User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Disallow: /search
Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

I don’t think the above file details are wrong but I don’t understand why I received such dangerous notification.

• How can I fix this issue?

Solution 1

If you have a message saying that "Google couldn't crawl your site because we were unable to access the robots.txt file". Then it is not the contents of the robots.txt file that is in question, it is that Google simply couldn't access the file. And when Google can't access a robots.txt file then it won't crawl the site.

Using fetch as Googlebot in Webmaster Tools is a good idea. If your robots.txt file fetches successfully then it could be a past issue. If not, then you obviously need to look further to ensure Googlebot access.

Solution 2

There is no such official command as Allow in robots.txt. By default, everything is allowed. (However, it is possible to use Allow to give exceptions when you are disallowing many directory paths in one route. Often, there is no requirement for this though).

Not that I would expect it to cause an issue however.

There is no reason to specify the Mediapartners-Google user agent either as this too, is just saying allow the crawling of everything.

All your robots.txt needs from the above is the following:-

User-Agent: *
Disallow: /search/

User-agent: Mediapartners-Google
Disallow: /

Sitemap: http://latest-seo-news-updates.blogspot.com/feeds/posts/default?orderby=UPDATED

Google Webmaster Tools will report a warning to say X amount of URL's on your site were blocked by your robots.txt if you are disallowing bots to any part of your site, in which case you are at /search/. You can expand this notice to view specifically what URL's were blocked and you may well find that it is only the ones you want disallowed that Google Webmaster Tools is warning about.

You can also run an application such as Xenu to crawl your site and establish what URL's can be crawled specifically. You can also fetch as Googlebot and test your robots.txt file from within Google Webmaster Tools that will alert you to any further issues or at least complete details of any issues.

Edit: Upon further clarification, added Disallow directive for UA Mediapartners-Google.

Comments

[Marian Popovych]

1. "User-agent: Mediapartners-Google Disallow:" What does this section mean? 2. "Google couldn't crawl your site because we were unable to access the robots.txt file." Probably, it means your server did not return your robots.txt file while Google had tried to get it.

[Sathiya Kumar V M]

It only disallow the media-partners of Google like adsence.. Does it is wrong?

[Marian Popovych]

As for me, you should write User-agent: Mediapartners-Google* Disallow: But it is not a thing. If I were in your shoes, I would check server logs and look for Goolgebot queries to /robots.txt

[Zistoloen]

Did you check encoding of your file? UTF-8 is recommended by Google.

[MrWhite]

@Marian: You should not include an asterisk on the end of the user agent (this is not a wildcard operator in this context - the lone * is a special case). John: Disallow: by itself (without a path element) does not disallow the robot - this directive will be ignored, so it will in fact allow the robot!

[MrWhite]

Sorry, I deleted my comment! ... It would seem from the comments above that the OP does actually want to disallow the Mediapartners-Google robot, so the Disallow: directive (without a path element) in the question would seem to be incorrect. It should read Disallow: / (with a slash).

[zigojacko]

Agreed - answer updated. Thanks for pointing out clarification as per comments.