TwitteR, ROAuth and Windows: register OK, but certificate verify failed

Question

I'm trying to get the number of followers of a large number of Twitter users with twitteR. Many of the other questions posted have been very useful in getting me this far, but none seem to be directly relevant to my problem, so far as I can see.

I can register my OAuth credentials to a twitter R session, but then I can't seem to do anything at all, I just get is this message:

Error in function (type, msg, asError = TRUE) : SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify fail

When I use the twitteR functions without using OAuth they work fine with no errors or warnings, but I bump into limits and protected accounts, which I thought I could avoid with OAuth.

Here are the details:

library(twitteR)
library(ROAuth)
library(RCurl)
#
# Here's how I register my credentials
#
requestURL <-  "https://api.twitter.com/oauth/request_token"
accessURL =    "https://api.twitter.com/oauth/access_token"
authURL =      "https://api.twitter.com/oauth/authorize"
consumerKey =   "------------"
consumerSecret = "-----------"
twitCred <- OAuthFactory$new(consumerKey=consumerKey,
                             consumerSecret=consumerSecret,
                             requestURL=requestURL,
                             accessURL=accessURL,
                             authURL=authURL)
download.file(url="http://curl.haxx.se/ca/cacert.pem",
              destfile="cacert.pem")
twitCred$handshake(cainfo="cacert.pem")
To enable the connection, please direct your web browser to: 
https://api.twitter.com/oauth/authorize?oauth_token=xxxx
When complete, record the PIN given to you and provide it here: xxxxxx
registerTwitterOAuth(twitCred)
[1] TRUE
# so the OAuth bit appears to be ok...
#
# save it for a future sessions...
save(list="twitCred", file="twitteR_credentials")
# works, in future I can just
load("twitteR_credentials")
registerTwitterOAuth(twitCred)
#
# try to get follower numbers, here's where it goes south
me <- getUser("Rbloggers")
me$followersCount
Error in function (type, msg, asError = TRUE)  :
SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
#
# another method, same problem
getUser("Rbloggers")$followersCount
Error in function (type, msg, asError = TRUE)  : 
SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
# 
# Here are the packages and versions I'm using
sessionInfo() 
R version 2.14.1 (2011-12-22)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252   
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Australia.1252    

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] ROAuth_0.9.2      digest_0.5.1      twitteR_0.99.19   rjson_0.2.6       RCurl_1.91-1.1   
 [6] bitops_1.0-4.1    igraph_0.5.5-4    topicmodels_0.1-4 tm_0.5-7          slam_0.1-23      
[11] modeltools_0.2-18 lasso2_1.2-12    

loaded via a namespace (and not attached):
[1] tools_2.14.1

How can I get the twitteR functions working after I register my credentials?

UPDATE: Trying @Btibert3's suggestion gives the same error:

> ## Authenticate with Twitter = this is an important peice of code
> registerTwitterOAuth(cred)
[1] TRUE
> ##########################################################################
> ## lets test out what our session limits look like
> ##########################################################################
> rate.limit <- getCurRateLimitInfo()
Error in function (type, msg, asError = TRUE)  : 
  SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

UPDATE following @flz's suggestion to add cainfo="cacert.pem" everywhere has fixed my problem:

rate.limit <- getCurRateLimitInfo( cainfo="cacert.pem")
rate.limit
                             resource limit remaining               reset
1                  /lists/subscribers   180       180 2013-03-27 09:35:37
2                         /lists/list    15        15 2013-03-27 09:35:37
3                  /lists/memberships    15        15 2013-03-27 09:35:37
4                   /lists/ownerships    15        15 2013-03-27 09:35:37
5                /lists/subscriptions    15        15 2013-03-27 09:35:37
6                      /lists/members   180       180 2013-03-27 09:35:37
7             /lists/subscribers/show    15        15 2013-03-27 09:35:37
8                     /lists/statuses   180       180 2013-03-27 09:35:37
9                         /lists/show    15        15 2013-03-27 09:35:37
10                /lists/members/show    15        15 2013-03-27 09:35:37
11     /application/rate_limit_status   180       179 2013-03-27 09:35:37 (etc)

Session Info:

sessionInfo()
R version 2.15.3 (2013-03-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ROAuth_0.9.2   digest_0.6.3   twitteR_1.1.0  rjson_0.2.12   RCurl_1.95-4.1 bitops_1.0-5  

loaded via a namespace (and not attached):
[1] tools_2.15.3

Solution 1

Try:

getUser("Rbloggers")$followersCount
Error in function (type, msg, asError = TRUE)  : 
  SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
getUser("Rbloggers",cainfo="cacert.pem")$followersCount
[1] 2752

Every get/update action needs a cainfo="cacert.pem" append behind. That annoy.

Solution 2

I had received the error you described above in the past, but this has worked for me.

#=======================================================================================
## ON windows, we need to dowload the certificate for OAUTH
## NOTE:  you will need to setup an app on Twitter
## dev.twitter.com <- get your KEY/SECRET
#=======================================================================================


##########################################################################
## Load packages
##########################################################################

library(twitteR)
library(ROAuth)

## set the directory
setwd("~/your/directory/here")


## Windows users need to get this file
download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")



##########################################################################
## Authenticate with Twitter
##########################################################################

## authenticate with the API
## requires that you have registered an app
KEY <- "KEY"
SECRET <-"SECRET"


## create an object that will save the authenticated onbject -- we can for later sessions
## will need to navigate to website and type in data to generate the file
## NOTE:  Only need to do this part once!!!
cred <- OAuthFactory$new(consumerKey = KEY, 
    consumerSecret = SECRET,
    requestURL = "https://api.twitter.com/oauth/request_token", 
    accessURL = "https://api.twitter.com/oauth/access_token", 
    authURL = "https://api.twitter.com/oauth/authorize")
cred$handshake(cainfo="cacert.pem")


## load the cred object in later sessions and simply pass to the registerTwitterOAuth
## After this file is saved, you only need to load the cred object back into memory
save(cred, file="twitter authentication.Rdata")


## Authenticate with Twitter = this is an important peice of code
registerTwitterOAuth(cred)


##########################################################################
## lets test out what our session limits look like
##########################################################################
rate.limit <- getCurRateLimitInfo()

## If return 350, Authenticated session = more API calls allowed / hour
rate.limit$hourlyLimit
rate.limit$remainingHits
rate.limit$resetTime

Solution 3

UPDATE This was an interim solution, see @flz's answer and my edited question for the final solution.

Since I posted the question I've been using this simple loop as a workaround (proper solutions are still welcome!). It's going to take a long time, but at least it'll give me the data. Perhaps someone else will find it useful also.

# load library
library(twitteR)
#
# Search Twitter for your term
s <- searchTwitter('#rstats', n=1500) 
# convert search results to a data frame
df <- do.call("rbind", lapply(s, as.data.frame)) 
# extract the usernames
users <- unique(df$screenName)
users <- sapply(users, as.character)
# make a data frame for the loop to work with 
users.df <- data.frame(users = users, 
                       followers = "", stringsAsFactors = FALSE)
#
# loop to populate users$followers with follower 
# count obtained from Twitter API
for (i in 1:nrow(users.df)) 
    {
    # tell the loop to skip a user if their account is protected 
    # or some other error occurs  
    result <- try(getUser(users.df$users[i])$followersCount, silent = TRUE);
    if(class(result) == "try-error") next;
    # get the number of followers for each user
    users.df$followers[i] <- getUser(users.df$users[i])$followersCount
    # tell the loop to pause for 60 s between iterations to 
    # avoid exceeding the Twitter API request limit
    print('Sleeping for 60 seconds...')
    Sys.sleep(60); 
    }
#
# Now inspect users.df to see the follower data

Solution 4

There seems to be a problem with ROauth 0.9.1, if you use that version you should take a look at this post by the developer of twitteR (it has worked for me): http://lists.hexdump.org/pipermail/twitter-users-hexdump.org/2012-March/000075.html

Solution 5

I'm not sure if I'm just not seeing this right, but I'm thinking it would cause an issue that your file is saved to cacert.perm but you're tell the handshake to look in cacert.pem?

> download.file(url="http://curl.haxx.se/ca/cacert.pem",
>              destfile="cacert.perm")
> twitCred$handshake(cainfo="cacert.pem")

View more solutions

Comments

[Ben]

thanks for sharing your script, it doesn't seem to make any difference for me... perhaps it's a problem beyond R?

[Btibert3]

when you run the line that is registerTwitterOAuth(cred), do you return a False?

[Btibert3]

regardless, even if you do not authenticate, you can still query a good number of data elements from the Twitter API.

[Ben]

Yes, registerTwitterOAuth() returns TRUE using your method and my method. As an alternative, I have a loop with a 60 s sleep interval running at the moment. Probably take a couple of days to get what I want though!

[Ben]

You can see in my sessionInfo above that I'm already using 0.9.2

[Ben]

Thanks for the suggestion. I've given this a try and it doesn't seem to have made any difference.

[Simon O'Hanlon]

@Ben Can you paste your sessionInfo()? Might help diagnose?

[Ben]

This seems to have done it, thanks. When I do s <- searchTwitter('#caa2013') I get the error, but when I do s <- searchTwitter('#caa2013', cainfo="cacert.pem") it works fine.

[Ben]

Good catch, definitely a typo there! But I don't think that was the main issue.

[Ben]

Sure, I've popped it in the Q.

[zap2008]

just to add to what @Ben said, this solution didn't work for me until I updated cacert.pem by following that link and saved it in my working directory.

[Ricardo]

@zap2008 Where should I place cacert.pem file?

[Ricardo]

@zap2008 does not work either, I finally disable security with curl_setopt($feed, CURLOPT_SSL_VERIFYHOST, 0) and curl_setopt($feed, CURLOPT_SSL_VERIFYPEER, 0);

[RRR]

if the limit is not enough.. create 10 users and RDS token file. (Switch between tokens). With that you can reach 3M ids per hour. Or contact with the Twitter DEV team. But I think, 3M/ hour far enough. ;)