Wordpress remove Robots Meta Tag noindex

Question

were experiencing a strange issue with a wordpress sites meta robots tag. All pages have the following meta tag and we cant seem to remove it

<meta name="robots" content="noindex,follow"/>

We have unchecked "Discourage search engines from indexing this site" in Settings > Reading > Search Engine Visibility but it does nothing.

We are using the Yoast SEO plugin but even when this is disabled the still remains. In fact, we have tried disabling all plugins to check nothing was interfering with it.

We have setup our Robots.txt file as follows:

User-agent: *
Disallow: 
Sitemap: http://speysidedistillery.co.uk/sitemap.xml

Im not sure if the Robots.txt takes precedent over the robots meta tag or not and there doesn't seem to be a definitive answer, as far as i can gather the most restrictive one will take precedent i.e in our case the meta tag.

This is mainly giving us issues with our google listing with the warning "A description for this result is not available because of this site's robots.txt" appearing instead of our sites description.

If worse comes to worst we can edit the wp_no_robots function in wp-includes/general-templates.php but would prefer to resolve this without editing the wp core files.

Any light anyone could shone on this would be great as we are at a loss, cheers

The site can be found at http://speysidedistillery.co.uk/

Solution 1

i Developed an E commerce site for a client and i was stacked with this issue for more than a week, all pages were indexed except for the Home Page.

After a long debugging of Yoast Seo Plugin, i got a solution for you guys.

SOLUTION The Homepage has a search page on it (for searching for products). By default, Yoast disables 'index' meta tag for Search Page and 404 Page. funny enough if your homepage has a search page or search functionality on it, your site may not be indexed by google or any other search engine.

To fix this default by Yoast SEO Plugin, follow the following steps:

STEP 1: Log into your cpanel and navigate to /public_html/yourwebsite.com/wp-content/themes/nameOfTheTheme/functions.php add this line of code to your theme's function.php file (at the top of the page)

 /**
  * EDITED BY OBOT ERNEST
  * REMOVES NOINDEX META TAG FROM HOME PAGE  & SEARCH PAGE IMPLEMENTED BY YOAST
 */


    add_filter('wpseo_robots', 'yoast_no_home_noindex', 999);
    function yoast_no_home_noindex($string= "") {
        if (is_home() || is_front_page() || is_search()) {
            $string= "index,follow";
        }
        return $string;
    }

After pasting the code, save and close file.

STEP 2 Note this second step is for those that are using any of the following SEO plugins (Yoast Premium, Yoast Free or wordpress-seo-premium plugin)

Navigate to: /public_html/youwebsite.com/wp-content/plugins/wordpress-seo-premium/frontend/class-frontend.php

replace the code at line 713 with this one below here:

    if ( is_search() ) {
$robots['index'] = 'index';
}
    if ( is_404() ) {
$robots['index'] = 'noindex';
}

Save file and close

Congratulations! you just made it. It working perfectly guys!

Solution 2

If you are using Roots Bedrock, then check the value of WP_ENV in your .env file. If it is set to development then this automatically overrides the value of blog_public to always be false, meaning the "no robots meta" is always injected.

Simply change it to WP_ENV='production' in your .env file, save the changes and it will be fine.

Solution 3

I found a solution that worked for me here

add_action( 'init', 'remove_wc_page_noindex' );
function remove_wc_page_noindex() {
    remove_action( 'wp_head', 'wc_page_noindex' );
}

This question appears to be on three threads:

1. How do I stop Wordpress from inserting a noindex meta tag?
2. Wordpress remove Robots Meta Tag noindex
3. Cannot remove action noindex in wordpress

Solution 4

Navigate to Yoast SEO -> Search Appearance. Choose tab Content Type or Taxonomies depending on the page you need indexed by google.

In my case, google can not index my tags page (/tag/tag-name)), So i switched Show Tags in search result? to Yes. It will change robots meta tag from:

<meta name="robots" content="noindex, follow" />

to

<meta name="robots" content="index, follow" />

Hope can help.

View more solutions

Comments

[plasticinsect]

Allowing a site in robots.txt does not prevent pages from being blocked by robots meta tags. They are different things. Robots.txt tells the crawler whether or not to load the page in the first place. Robots meta tags tell search engines whether to index (or follow links on) pages that they have already loaded.

[Christopher Craig]

Is this possibly for an older version of WP? Were using 4.0. As far as i can see the privacy setting are under Settings > Reading. I cant see an Options or pPrivacy tab anywhere

[Davit Huroyan]

Yes you are right :) it was in older vresions in 4 version settings > reading Discourage search engines from indexing this site

[Suraj Lulla]

is this still working?

[ThaoD5]

You're a life saver, I just spent 3 hours looking at every single plugin / file and it was in the fact the .env set to development ... Thanks a lot