How to resolve "Google can't find your site's robots.txt" error?

Question

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows "Google can't find your site's robots.txt".

What can I do to resolve this issue? Before this issue arose, my site was indexed within a few mintues, but now I find that it took time to be indexed in Google's search.

When I access http://mydomain.com/robots.txt, it shows the data below:

User-agent:

*Disallow: /wp-admin/

Disallow: /wp-includes/

I found Blocked URLs = 0, also no any other errors.

Is there any other thing I need to change? Or what could be the solution for this? Any help would be appreciated.

Solution

Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

Also, that was probably just a typo, but your robots.txt should look like this:

User-agent: * 

Disallow: /wp-admin/
Disallow: /wp-includes/

Comments

[MrWhite]

Is this a recurring error in GWT? Check the date. I had the same error on 30th Sept on one of my sites. However, the same error displays whenever I go into GWT which makes it look as if it is persistent, but it is the same (old) message that is displayed.

[stealthyninja]

If you see it's an old message, simply go to Webmaster Tools → All Messages, find the offending message and tick the check box to the left of it to activate the delete button.

[MrWhite]

Where is this Settings > Privacy option you mention? I do not see this in GWT?

[stealthyninja]

@w3d - It's not in GWT, it's in WordPress which is the software the OP is using for his site.