How to Edit or Remove robots.txt on a WordPress powered website
Solution 1
You need to remove both lines from your robots.txt
file. The robots file is located in the root directory of your web hosting folder, this normally can be found in /public_html/
and you should be able to edit or delete this file using:
- FTP using a FTP client such as FileZilla or WinSCP
- SFTP (SSH File Transfer Protocol) using a SFTP client such as FileZilla or WinSCP
- SSH using a SSH client such as Putty
- WebDev (Web Disk) using a compatible OS that allows mapping of WebDav Drives.
- WordPress using a robots.txt WP plugin
Solution 2
If this is fresh install of Wordpress it could because you have set the privacy settings to stop search engines from crawling the site.
(and there will be no physical robots.txt on the server, as wordpress creates it on the fly)
Go to your settings in wordpress and see if this box is ticked:
If so, uncheck it and then the robots.txt should change to
User-agent: *
Disallow:
If you still have issues and the robots.txt is still set to block crawlers, then explore the other option as outlined by Facet.
Related videos on Youtube
Facet
Updated on September 18, 2022Comments
-
Facet over 1 year
When using Google's fetch test on my WordPress site, it reports that my robots.txt is blocking pages and resources from being indexed.
Google reports that my Robots.txt looks like this:
User-agent: * Disallow: /
How can I resolve this issue?
- Where can I find robots.txt?
- How do I edit or remove this file?
-
Duarte Patrício over 7 yearsHe didn't mention he had Yoast installed.
-
RamboNo5 about 6 yearsThanks! That was my problem. I manually configured Apache to send an X-Robots-Tag with noindex in the http header. See this wordpress ticket: Search Engine Visibility option does not work as intended