How to crawl twitter tweet information without OAuth authentication?

Question

I'm required to crawl twitter and analyze the tweets for information. I figured the best way would be to use the search API, however it seems that now the api requires the OAuth authentication. Would registering as a developer be the only way? Are there alternatives?

Solution 1

Lots of misinformation here. You do not have to screen scrape. You do not have to register an app to do this. There are no API keys in Twitter. You do not have to use any authorization to read data from the search API. The rest of the API requires OAuth, but not search.

To use the search API you can just make a request against the following URL: http://search.twitter.com/search.json?q=[keywords]

For example to search for pizza: http://search.twitter.com/search.json?q=pizza

You get JSON data back that you can read in any program. If you use PHP, you can use cURL to make the request and json_decode() to convert the result into an object you can iterate through in a foreach() loop.

Solution 2

The (chosen) answer of Adam Green is not working anymore since Twitter closed their REST API v1.0.

If you call the given example to query for pizza:

http://search.twitter.com/search.json?q=pizza

You get the following error message: (in json, though ;)

{"errors": 
   [{"message": "The Twitter REST API v1 is no longer active. 
      Please migrate to API v1.1. https://dev.twitter.com/docs/api/1.1/overview",
     "code": 64}
   ]
}

You can check the API 1.1 search spec here

Another change from 1.0 to 1.1 is that all 1.1 API Calls require an authentication including search.

Solution 3

If you want to analyse large amounts of tweets, you should be using the Streaming API. You will need to register for access to this.

You could also use the Search API for which you do not need to register. But this is rate limited.

Solution 4

You could try Python or cURL.

View more solutions

Comments

[goh]

I'm trying my best not to screen-scrape. So I'm required to register right?

[goh]

may i know what is the difference between the streaming api and the rest api?

[goh]

this certainly clears things up! May I know what's the difference between the streaming and the rest api there? And do i know authorization for both?

[Adam Green]

The streaming API is for tracking tweets by keyword, user id or location in real time. It still uses basic authentication, which is the account name and the password. The Rest API is for getting account info, like the list of followers, or performing an action, like sending a tweet. The Rest API uses OAuth for authentication.

[omarzouk]

REST API is no longer active.