Twitter4j - Rate limit exceeded

Question

I would like to get followers using getFollowersIds() in Twitter4j, but I get

ConnectionErrorException ... rate limit exceeded

public static void main(String[] args) {
        try {
            Twitter twitter = TwitterFactory.getSingleton();
            String[] srch = new String[]{"TechCrunch"};
            ResponseList<User> users = twitter.lookupUsers(srch);
            for (User user : users) {

                UserHarvest us = new UserHarvest(6017542);
                us.getFollowersIds();
                try {
                    us.getContributors();
                } catch (ConnectionErrorException ex) {
                    Logger.getLogger(UserHarvest.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        } catch (TwitterException ex) {
            Logger.getLogger(UserHarvest.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

Error message:

Exception in thread "main" harvest.twitterharvest.ConnectionErrorException: Connection could not have been established
    at harvest.twitterharvest.WrapperTwitter4J.getFollowersIDs(WrapperTwitter4J.java:75)
    at harvest.twitterharvest.UserHarvest.getFollowersIds(UserHarvest.java:106)
    at harvest.twitterharvest.UserHarvest.main(UserHarvest.java:140)
Caused by: 429:Returned in API v1.1 when a request cannot be served due to the application's rate limit having been exhausted for the resource. See Rate Limiting in API v1.1.(https://dev.twitter.com/docs/rate-limiting/1.1)
message - Rate limit exceeded
code - 88

Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=92c30ec6 or
    http://www.google.co.jp/search?q=19e1da11
TwitterException{exceptionCode=[92c30ec6-19e1da11], statusCode=429, message=Rate limit exceeded, code=88, retryAfter=-1, rateLimitStatus=RateLimitStatusJSONImpl{remaining=0, limit=15, resetTimeInSeconds=1384512813, secondsUntilReset=904}, version=3.0.4}
    at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:162)
    at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61)
    at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:89)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1894)
    at twitter4j.TwitterImpl.getFollowersIDs(TwitterImpl.java:409)
    at harvest.twitterharvest.WrapperTwitter4J.getFollowersIDs(WrapperTwitter4J.java:73)
    ... 2 more

I see secondsUntilReset=904. I run code 1 hour later and get the same error message. I don't know why. Thanks for any help.

Solution 1

There are rate limits in the Twitter API. You cannot call a given Twitter API endpoint more than a given number of times per 15 minutes (on behalf of the authencated user or not).

What happened to you is that your code must have reached the rate limit very quickly (the endpoint to retrieve followers IDs is limited to 15 calls per 15 minutes for a given authenticated user) so you will have to wait (904 seconds) before trying again.

Be careful to the Twitter API endpoints you are calling (through Twitter4J) in order to economize your API calls and thus avoid reaching the rate limit.

For further informations, have a look at those resources on Twitter Developers, especially at its documentation :

• REST API Rate Limiting in v1.1 (general topic about rate limiting in the Twitter API)
• REST API v1.1 Limits per window by resource (rate limits for each endpoint)
• Error Codes and Responses (which explains to you what is your '88' response code).

Solution 2

You dont have to wait to fix rate limit issue. If you have many tokens corresponding different twitter accounts, then you can renew your current used token, right. Think about the big picture. Let's say that you have 25 different accounts(tokens) in a list or array object, and you are getting one of them each time that you have a rate limit issue. Then when you reach to the last item in list token list, then you can refill this list with these 25 token accounts, right. The beauty is this, if you have good enough number of accounts(tokens), each refilling loop, the system automatically will be waited to recover the used accounts, so you will not have to worry about the rate limit and your application will never stop because of rate limit issue. I have already implemented this mechanism in my project and it works perfect. This is the only best solution.

Comments

[Matt]

I know there are limits in the Twitter API, but I don't understand why I get the same error message one hour later and in the meanwhile I haven't used Twitter API services.

[air-dex]

How many users do you have in your users list? As I said in my answer, you can only make 15 requests per 15 min to retrieve followers IDs. So if your users list has got at least 16 users (15 + 1), the 16th member will make a 16th request in less than 15 minutes to the GET followers/ids endpoint. The Twitter API will consequently respond 429 and the exception will be raised. This scenario will happen each time to will try to execute your code (if the users list has got > 15 members, of course), even one hour after the previous execution.

[Matt]

I have only one user in list. As you can see "TechCrunch" in example

[air-dex]

I don't think that you understand what your code really does. First it searches "TechCrunch" users on Twitter. Then it retrieves the followers IDs and the contributors of each and every one of them. And it looks like that your reseach returns you far more than one single user. I did the research on the Web interface ( twitter.com/search?q=TechCrunch&src=typd&mode=users ) and it returns me more users to reach the rate limit than required: @TechCrunch (of course!), @TCFR, @TechCrunchOnion, @TechCrunchTV, @RSS_TechCrunch, @TechCrunchItaly, @TechCrunchIndia, @TC_Europe, @alexia...

[nullwriter]

Hey man, how would you check for a 429 TwitterException during a loop to then change the tokens of the twitter object?

[mgokhanbakal]

Actually, I was checking the error message whether it contains "Rate limit exceeded" or not. If it contains, I renew the current token. It was the only way at that time that I used. Maybe, now twitter4J has a machanism for that. I dont know.

[user3764893]

@MehmetGokhanBAKAL are you still here? I have a question.

[mgokhanbakal]

Sure, You can ask your question.

[user3764893]

The solution you have mentioned above would make sense for a single user, using that app. But what if there are 10.000 user using that app at the same time? With only 25 accounts(tokens) that would be unhandleable, right?

[mgokhanbakal]

Yes, that is correct. However, my application is not a public application. What I mean is I am the only user for the application just for academic purposes. :D If you wanted to develop a publicly available application which would be used by many users, it would be unhandleable. :D

[Ravindra S]

@MehmetGokhanBAKAL How did you create 25 different tokens? I am assuming it means 25 different accounts. To do that every new account needs a different mobile number. If you use the same mobile number twitter says "The phone number you're trying to use is already associated with another Twitter account.". I don't know how you have achieved this. Please explain.

[mgokhanbakal]

The old policy was different. When I created more than one account, there was no obligation like providing a valid phone number to create an account. So, that is why I was able to create many accounts and tokens.

[A. Harkous]

These links has changed now, here are the updated ones: rate limits: developer.twitter.com/en/docs/basics/rate-limits.html, response codes: developer.twitter.com/en/docs/basics/response-codes