Block a site from search engine - DuckDuckGo
Solution 1
DuckDuckGo should honour your robots.txt
. Their bot DuckDuckBot
is documented at https://duckduckgo.com/duckduckbot.
But note: the DuckDuckGo bot isn’t crawling everything itself (as DuckDuckGo gets results from other sources), so your pages might still show up if you don’t block the bots of these other sources (like Bing). Refer to mlissner’s answer for more details.
With robots.txt
, there are two things to consider:
- It takes time until changes in your
robots.txt
are recognized. You have to wait until the relevant bot visits your site again. - Even if your URLs are blocked in the
robots.txt
, search engines may still list your URLs in their search results (without crawled metadata like title and description).
Using the robots
-meta
element with noindex
would prevent even listing the URLs in search engines like Google, but DDG doesn’t seem to support it.
Note that you used wrong quotation marks in your example. It should be
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
instead of
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
Solution 2
DuckDuckGo is an odd duck when it comes to inclusion in their results. I've done a fair bit of research on this topic across a number of search engines and have had some email back and forth with DDG.
Here's the deal. They get their content from other search engines, as listed here. To my knowledge their search results don't indicate which search engine was its source, so for your content to be removed you need to basically go upstream to all of their sources and get your content removed from there. If that sounds onerous, don't worry — you'd want to do that anyway, right?
DDG does have its own crawler as well, aptly called the DuckDuckBot. It does not honor the noindex
HTML tag, nor the HTTP header (it does honor robots.txt
), but that doesn't seem to matter because no new results are created by the DuckDuckBot. To my knowledge, this isn't documented anywhere, but I spoke with their staff, which I quote below:
DDG says (2014-06-06):
We get our results from multiple sources and our own crawler wouldn't be the cause of your [problem]. Our crawler only does very specific tasks, like looking (and not actually crawling) parked domains, spam sites, etc.
If there are results from [your website] appearing on DuckDuckGo and shouldn't be, they're likely flowing from one of our upstream sources. If removed there, then they'll stop showing in our results.
I respond:
OK, so nothing gets put in your index via your crawlers, which indeed do not support
noindex
HTML or HTTP tags?
They confirm:
Yep! Sorry for the confusion and, if you see anything out of the ordinary, please feel free to let us know.
So then the only remaining question is how do you remove your content from the upstream providers. For that, I point you to my blog since it differs by provider. The crux of it is:
- Use
noindex
HTML meta tag andx-robots
HTTP tag (for images and such) to tell search engines not to include something in their results; - List your entire website in your sitemap.xml file so that all search engines can find it there.
- Use
robots.txt
to block the search engines that do not supportnoindex
orx-robots
tag.
And for bonus points:
- Set your
sitemaps.xml
files so they havenoindex
set up (and thus won't show up in search results). - Do likewise for your
robots.txt
file.
It's a complicated world.
Vimalnath
Interests: PHP Javascript Jquery Oracle Drupal PayPal Integration http://www.techycommerce.com
Updated on July 21, 2022Comments
-
Vimalnath almost 2 years
I have a development site https://text-domain.com. (not a real site) When I go to https://duckduckgo.com and search for text-domain.com, it does return results.
What have I tried so far:
Created
robots.txt
file with following code(put in in my root directory i.e in text-domain.com/robots.txt):User-agent: * Disallow: /
Then added meta-tag like this in my template file:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Even after doing this, I searched on DuckDuckGo and it yielded the same result. Any suggestions would be welcome.
P.S
Hi, after waiting for few days there are 2 findings:
- Still, the search results are fetched.
-
But I see an message for that result saying : "We would like to show you a description here but the site won't allow us."
Is it possible to completely block from showing in the results?
-
Vimalnath over 10 yearsThanks for that link, however I quotation marks looked correct in my actual site, so guess I need to wait for sometime to get my problem resolved.
-
Vimalnath over 10 yearsHi, after waiting for few days there are 2 findings: -Still, the search results are fetched. - But I see an message for that result saying : "We would like to show you a description here but the site won't allow us." , is it possible to completely block from showing in the results?
-
unor over 10 years@vimal: Then DDG doesn’t seem to interpret
meta
-robots
in the same way as Google. Someone says that they don’t support themeta
-robots
at all (at least in 2012). -
mlissner almost 10 yearsNote that the DuckDuckBot only does maintenance-type tasks and blocking it will not get your stuff out of DDG. See my answer for more detail.