Get all past tweets via Twitter Streaming API

Question

How can i get all past tweets via Twitter Streaming API? As you may know it sends real-time tweets, not past ones. Useful count parameter was disabled in 2010. REST API have so bad limitations that it would take a whole life to get all past tweets. Is there any solution?

Solution

Sorry, you can't get historical tweets from the streaming API unless you have elevated access (e.g., Firehose, Retweet, Link, Birddog or Shadow). You can request additional access by emailing api at twitter dot com. Note that you cannot use the track parameter with count.

Only the last 150,000 statuses sent through Twitter are available when using count. To quote the Twitter docs:

On unfiltered streams, all considered statuses are delivered, so the number requested is the number returned. On filtered streams, the number requested is the number of statuses that are applied to the filter predicate, and not the number of statuses returned.

Also, that circular buffer containing the last 150,000 tweets is subject to the rate in which they are flowing out of Twitter. So, if the current tweet rate is 600 tweets per second, the buffer only holds the last 250 seconds of data, and if you are interested in just a few users, there probably won't be more than just a couple tweets in the buffer, if any.

If you're authenticated, the REST API will allow you to pull 1,680,000 tweets per day, but the real limiting factor is that you can only get at 3,200 tweets per user. The search API is worse, since it currently seems to be returning only about four days worth of tweets from a specific target.

Note: Edited after abraham brought up count is still available with elevated access to the streaming API.

Comments

[clumpter]

1,680,000 tweets per day? Where did you get that number from?

[arcain]

@clumpter 350 requests/hour * 24 hours/day * 200 tweets/request = 1,680,000 tweets/day (when fetching from statuses/user_timeline which allows 200 tweets/request).