duckduckgo API not returning results

Question

Edit I now realize the API is simply inadequate and is not even working. I would like to redirect my question, I want to be able to auto-magically search duckduckgo using their "I'm feeling ducky". So that I can search for "stackoverflow" for instance and get the main page ("https://stackoverflow.com/") as my result.

I am using the duckduckgo API. Here

And I found that when using:

r = duckduckgo.query("example")

The results do not reflect a manual search, namely:

for result in r.results:
    print result

Results in:

>>> 
>>> 

Nothing.

And looking for an index in results results in an out of bounds error, since it is empty.

How am I supposed to get results for my search?

It seems the API (according to its documented examples) is supposed to answer questions and give a sort of "I'm feeling ducky" in the form of r.answer.text

But the website is made in such a way that I can not search it and parse results using normal methods.

I would like to know how I am supposed to parse search results with this API or any other method from this site.

Thank you.

Solution 1

If you visit DuckDuck Go API Page, you will find some notes about using the API. The first notes says clearly that:

As this is a Zero-click Info API, most deep queries (non topic names) will be blank.

An here's the list of those fields:

Abstract: ""
AbstractText: ""
AbstractSource: ""
AbstractURL: ""
Image: ""
Heading: ""
Answer: ""
Redirect: ""
AnswerType: ""
Definition: ""
DefinitionSource: ""
DefinitionURL: ""
RelatedTopics: [ ]
Results: [ ]
Type: ""

So it might be a pity, but their API just truncates a bunch of results and does not give them to you; possibly to work faster, and seems like nothing can be done except using DuckDuckGo.com.

So, obviously, in that case API is not the way to go.

As for me, I see only one way out left: retrieving raw html from duckduckgo.com and parsing it using, e.g. html5lib (it worth to mention that their html is well-structured).

It also worth to mention that parsing html pages is not the most reliable way to scrap data, because html structure can change, while API usually stays stable until changes are publicly announced.

Here's and example of how can be such parsing achieved with BeautifulSoup:

from BeautifulSoup import BeautifulSoup
import urllib
import re

site = urllib.urlopen('http://duckduckgo.com/?q=example')
data = site.read()

parsed = BeautifulSoup(data)
topics = parsed.findAll('div', {'id': 'zero_click_topics'})[0]
results = topics.findAll('div', {'class': re.compile('results_*')})

print results[0].text

This script prints:

u'Eixample, an inner suburb of Barcelona with distinctive architecture'

The problem of direct querying on the main page is that it uses JavaScript to produce required results (not related topics), so you can use HTML version to get results only. HTML version has different link:

• http://duckduckgo.com/?q=example # JavaScript version
• http://duckduckgo.com/html/?q=example # HTML-only version

Let's see what we can get:

site = urllib.urlopen('http://duckduckgo.com/html/?q=example')
data = site.read()
parsed = BeautifulSoup(data)

first_link = parsed.findAll('div', {'class': re.compile('links_main*')})[0].a['href']

The result stored in first_link variable is a link to the first result (not a related search) that search engine outputs:

http://www.iana.org/domains/example

To get all the links you can iterate over found tags (other data except links can be received similar way)

for i in parsed.findAll('div', {'class': re.compile('links_main*')}):
    print i.a['href']

http://www.iana.org/domains/example
https://twitter.com/example
https://www.facebook.com/leadingbyexample
http://www.trythisforexample.com/
http://www.myspace.com/leadingbyexample?_escaped_fragment_=
https://www.youtube.com/watch?v=CLXt3yh2g0s
https://en.wikipedia.org/wiki/Example_(musician)
http://www.merriam-webster.com/dictionary/example
...

Note that HTML-only version contains only results, and for related search you must use JavaScript version. (vithout html part in url).

Solution 2

After already getting an answer to my question which I accepted and gave bounty for - I found a different solution, which I would like to add here for completeness. And a big thank you to all those who helped me reach this solution. Even though this isn't the solution I asked for, it may help someone in the future.

Found after a long and hard conversation on this site and with some support mails: https://duck.co/topic/strange-problem-when-searching-intel-with-my-script

And here is the solution code (from an answer in the thread posted above):

>>> import duckduckgo
>>> print duckduckgo.query('! Example').redirect.url
http://www.iana.org/domains/example

Comments

[Inbar Rose]

same result, nothing. the problem is that the r.results is an empty array, the API is returning no results at all.

[Inbar Rose]

r.related returns related searches/queries which is not what i am trying to get though... even though in some instances it could be usefull. obviously it is a sort of "duct-tape solution"

[couchemar]

if you try: api.duckduckgo.com/?q=example&format=xml&pretty=1 you get empty results too.

[Inbar Rose]

true, but obviously my code is not searching for "example" most everything else also returns no results as well.

[Inbar Rose]

thank you. this helps me understand what the problem is, where did you find that? :P i tried writing a parser for the regular html page of duckduckgo, but i was having problems because it uses java or something and the results didnt come out in proper html format...

[Rostyslav Dzinko]

It works fine for me with BeautifulSoup. Will update the answer

[Inbar Rose]

well, thats wrong, the result you get is from the related searches.

[Rostyslav Dzinko]

It's just an example of that the page is consistent HTML, you can do this way to get all another results

[Inbar Rose]

so using the html page, can i get more than just one result?

[Rostyslav Dzinko]

Yes, you can iterate over links, just look at BeautifulSoup documentation how to process parsed data.

[Inbar Rose]

could you post a new answer so i can accept it more concisely. which uses the HTML duckduckgo, and then returns a list of results.

[Rishi Dua]

The link seems to be dead

[Inbar Rose]

Yes, it seems so. Sorry - The main point of the thread I posted here anyway. Most of the rest was just a back and forth discussion about the problems.

[Mr Alexander]

is there any kind of TOS in duckduckgo (like in google) that prohibit scrapping their page?

[J. Doe]

is it ok to web scrap the search results of duckduckgo ? when open their robots.txt file, User-agent: * Disallow: /*?. In the other hand on link we can see that million of searches are made by bots ?

[J. Doe]

lets check : rp = urllib.robotparser.RobotFileParser(); rp.set_url("https://duckduckgo.com/robots.txt"); rp.read(); rp.can_fetch("*", "https://duckduckgo.com/?q=can we scrap duckduckgo?") returns True

[J. Doe]

however : rp.can_fetch("*", "https://duckduckgo.com/html/?q=can we scrap duckduckgo") returns False