POST request using RCurl

Question

As a way of exploring how to make a package in R for the Denver RUG, I decided that it would be a fun little project to write an R wrapper around the datasciencetoolkit API. The basic R tools come from the RCurl package as you might imagine. I am stuck on a seemingly simple problem and I'm hoping that somebody in this forum might be able to point me in the right direction. The basic problem is that I can't seem to use postForm() to pass an un-keyed string as part of the data option in curl, i.e. curl -d "string" "address_to_api".

For example, from the command line I might do

$ curl -d "Tim O'Reilly, Archbishop Huxley" "http://www.datasciencetoolkit.org/text2people"

with success. However, it seems that postForm() requires an explicit key when passing additional arguments into the POST request. I've looked through the datasciencetoolkit code and developer docs for a possible key, but can't seem to find anything.

As an aside, it's pretty straightforward to pass inputs via a GET request to other parts of the DSTK API. For example,

ip2coordinates <- function(ip) {
  api <- "http://www.datasciencetoolkit.org/ip2coordinates/"
  result <- getURL(paste(api, URLencode(ip), sep=""))
  names(result) <- "ip"
  return(result)
}
ip2coordinates('67.169.73.113')

will produce the desired results.

To be clear, I've read through the RCurl docs on DTL's omegahat site, the RCurl docs with the package, and the curl man page. However, I'm missing something fundamental with respect to curl (or perhaps .opts() in the postForm() function) and I can't seem to get it.

In python, I could basically make a 'raw' POST request using httplib.HTTPConnection -- is something like that available in R? I've looked at the simplePostToHost function in the httpRequest package as well and it just seemed to lock my R session (it seems to require a key as well).

FWIW, I'm using R 2.13.0 on Mac 10.6.7.

Any help is much appreciated. All of the code will soon be available on github if you're interested in playing around with the data science toolkit.

Cheers.

Solution 1

With httr, this is just:

library(httr)
r <- POST("http://www.datasciencetoolkit.org/text2people", 
  body = "Tim O'Reilly, Archbishop Huxley")
stop_for_status(r)
content(r, "parsed", "application/json")

Solution 2

Generally, in those cases where you're trying to POST something that isn't keyed, you can just assign a dummy key to that value. For example:

> postForm("http://www.datasciencetoolkit.org/text2people", a="Archbishop Huxley")
[1] "[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":44,\"end_index\":61,\"matched_string\":\"Archbishop Huxley\"},{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":88,\"end_index\":105,\"matched_string\":\"Archbishop Huxley\"}]"
attr(,"Content-Type")
                charset 
"text/html"     "utf-8" 

Would work the same if I'd used b="Archbishop Huxley", etc.

Enjoy RCurl - it's probably my favorite R package. If you get adventurous, upgrading to ~ libcurl 7.21 exposes some new methods via curl (including SMTP, etc.).

Solution 3

From Duncan Temple Lang on the R-help list:

postForm() is using a different style (or specifically Content-Type) of submitting the form than the curl -d command. Switching the style = 'POST' uses the same type, but at a quick guess, the parameter name 'a' is causing confusion and the result is the empty JSON array - "[]".

A quick workaround is to use curlPerform() directly rather than postForm()

r = dynCurlReader()
curlPerform(postfields = 'Archbishop Huxley', url = 'http://www.datasciencetoolkit.org/text2people', verbose = TRUE,
             post = 1L, writefunction = r$update)
r$value()

This yields

[1]
"[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":0,\"end_index\":17,\"matched_string\":\"Archbishop
Huxley\"}]"

and you can use fromJSON() to transform it into data in R.

Solution 4

The simplePostToHost function in the httpRequest package might do what you are looking for here.

Solution 5

I just wanted to point out that there must be an issue with passing a raw string via the postForm function. For example, if I use curl from the command line, I get the following:

    $ curl -d "Archbishop Huxley" "http://www.datasciencetoolkit.org/text2people
[{"gender":"u","first_name":"","title":"archbishop","surnames":"Huxley","start_index":0,"end_index":17,"matched_string":"Archbishop Huxley"}]

and in R I get

> api <- "http://www.datasciencetoolkit.org/text2people"
> postForm(api, a="Archbishop Huxley")
[1] "[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":44,\"end_index\":61,\"matched_string\":\"Archbishop Huxley\"},{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":88,\"end_index\":105,\"matched_string\":\"Archbishop Huxley\"}]"
attr(,"Content-Type")
                charset 
"text/html"     "utf-8" 

Note that it returns two elements in the JSON string and neither one matches on the start_index or end_index. Is this a problem with encoding or something?

View more solutions

Comments

[rtelmore]

Thanks for the help! Is there any reason that the key is 'a'? I tried 'name', 'text', and a bunch of other crap.

[rtelmore]

Correction: I tried using 'name', etc. in a different call. I tried using, e.g. postForm(api, string) and you need postForm(api, a=string).

[Noah]

Right, you need to provide a key=value pair. 'a' was completely arbitrary (it's just the first letter that came to mind). Any of those others works just as well (as a="string", name="string", etc. "a"="string" won't work.)

[Noah]

I'd guess this is actually something on the API end - this is the type of thing you'd expect to see if they handle URL encoded things strangely. You could try using URLencode() on your argument, but that may not actually help.

[Avinash]

Hi, I wish to do the same thing as you but my content-type is "text/plain" is it possible to pass this in postForm?