Get tweets of a public twitter profile

Question

I have a list of usernames on Twitter whose profiles are public. I wish to get "all the tweets" they have posted from the day they formed their profile. I checked Twitter4J examples on GitHub.
According to the Twitter API documentation, only the 20 most recent tweets are returned. Is there anyway I could perform my task?

Solution 1

To use Twitter4J to get all posts from a user you'll have to make your request over multiple pages..

Below code based of an example on GitHub

Twitter unauthenticatedTwitter = new TwitterFactory().getInstance();
//First param of Paging() is the page number, second is the number per page (this is capped around 200 I think.
Paging paging = new Paging(1, 100);
List<Status> statuses = unauthenticatedTwitter.getUserTimeline("google",paging);

Just loop and keep grabbing new pages until there are no new posts should work.

Solution 2

Here is how to get ALL tweets for a user (or at least up to ~3200):

import java.util.*;
import twitter4j.*;
import twitter4j.conf.*;

public static void main(String[] a) {

    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.setOAuthConsumerKey("YOUR KEYS HERE");
    cb.setOAuthConsumerSecret("YOUR KEYS HERE");
    cb.setOAuthAccessToken("YOUR KEYS HERE");
    cb.setOAuthAccessTokenSecret("YOUR KEYS HERE");

    Twitter twitter = new TwitterFactory(cb.build()).getInstance();

    int pageno = 1;
    String user = "cnn";
    List statuses = new ArrayList();

    while (true) {

      try {

        int size = statuses.size(); 
        Paging page = new Paging(pageno++, 100);
        statuses.addAll(twitter.getUserTimeline(user, page));
        if (statuses.size() == size)
          break;
      }
      catch(TwitterException e) {

        e.printStackTrace();
      }
    }

    System.out.println("Total: "+statuses.size());
}

Solution 3

If you read through the Twitter's Documentation, you can retrieve up to 200 tweets at a time if you specify "count=200" in your API request.

You can also use "page=x" to get different paginated results; you could keep doing this until you've retrieved every tweet the user has posted.

I'm not sure how your Java application would create this, but your requests would probably look like:

http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=1
http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=2
http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=3

... etc.

Keep in mind that these requests are rate-limited so you'd need to be careful not to exceed the limit.