How do I POST a buffer of JSON using libcurl?

Question

My C++ program currently invokes curl through a pipe (popen("curl ...")) to POST a file of JSON data to a web server. This has obvious performance limitations due to the need to save the JSON to a file and invoke curl in a subshell. I'd like to rewrite it to use libcurl, but it is not clear to me how to do this. The command line I pass to popen() is:

curl -s -S -D /dev/null -H "Content-Type: application/json" -X POST -d file-of-json http://server/handler.php

The JSON data (about 3K) is sitting in a buffer in RAM before I need to post it. I was expecting to use libcurl's CURLOPT_READFUNCTION to spool the buffer to libcurl (but I am open to alternatives), and CURLOPT_WRITEFUNCTION to capture the server's reply, similar to how I read the reply from popen's pipe.

All that seems straightforward. What is confusing is which combination of CURLOPT_POST, CURLOPT_HTTPPOST, CURLOPT_POSTFIELDS, CURLOPT_HTTPHEADER I need. I have read many posts on this subject (no pun intended), and none exactly match my scenario. Any suggestions?

[Note that I normally do not have any URL-encoded form fields, like this: http://server/handler.php?I=do&not=use&these=in&my=query]

Solution 1

There is example code for this here: http://curl.haxx.se/libcurl/c/post-callback.html

/***************************************************************************
 *                                  _   _ ____  _
 *  Project                     ___| | | |  _ \| |
 *                             / __| | | | |_) | |
 *                            | (__| |_| |  _ <| |___
 *                             \___|\___/|_| \_\_____|
 *
 * Copyright (C) 1998 - 2011, Daniel Stenberg, <[email protected]>, et al.
 *
 * This software is licensed as described in the file COPYING, which
 * you should have received as part of this distribution. The terms
 * are also available at http://curl.haxx.se/docs/copyright.html.
 *
 * You may opt to use, copy, modify, merge, publish, distribute and/or sell
 * copies of the Software, and permit persons to whom the Software is
 * furnished to do so, under the terms of the COPYING file.
 *
 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
 * KIND, either express or implied.
 *
 ***************************************************************************/ 
/* An example source code that issues a HTTP POST and we provide the actual
 * data through a read callback.
 */ 
#include 
#include 
#include 

const char data[]="this is what we post to the silly web server";

struct WriteThis {
  const char *readptr;
  int sizeleft;
};

static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *userp)
{
  struct WriteThis *pooh = (struct WriteThis *)userp;

  if(size*nmemb sizeleft) {
    *(char *)ptr = pooh->readptr[0]; /* copy one single byte */ 
    pooh->readptr++;                 /* advance pointer */ 
    pooh->sizeleft--;                /* less data left */ 
    return 1;                        /* we return 1 byte at a time! */ 
  }

  return 0;                          /* no more data left to deliver */ 
}

int main(void)
{
  CURL *curl;
  CURLcode res;

  struct WriteThis pooh;

  pooh.readptr = data;
  pooh.sizeleft = strlen(data);

  curl = curl_easy_init();
  if(curl) {
    /* First set the URL that is about to receive our POST. */ 
    curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/index.cgi");

    /* Now specify we want to POST data */ 
    curl_easy_setopt(curl, CURLOPT_POST, 1L);

    /* we want to use our own read function */ 
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);

    /* pointer to pass to our read function */ 
    curl_easy_setopt(curl, CURLOPT_READDATA, &pooh);

    /* get verbose debug output please */ 
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

    /*
      If you use POST to a HTTP 1.1 server, you can send data without knowing
      the size before starting the POST if you use chunked encoding. You
      enable this by adding a header like "Transfer-Encoding: chunked" with
      CURLOPT_HTTPHEADER. With HTTP 1.0 or without chunked transfer, you must
      specify the size in the request.
    */ 
#ifdef USE_CHUNKED
    {
      struct curl_slist *chunk = NULL;

      chunk = curl_slist_append(chunk, "Transfer-Encoding: chunked");
      res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
      /* use curl_slist_free_all() after the *perform() call to free this
         list again */ 
    }
#else
    /* Set the expected POST size. If you want to POST large amounts of data,
       consider CURLOPT_POSTFIELDSIZE_LARGE */ 
    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (curl_off_t)pooh.sizeleft);
#endif

#ifdef DISABLE_EXPECT
    /*
      Using POST with HTTP 1.1 implies the use of a "Expect: 100-continue"
      header.  You can disable this header with CURLOPT_HTTPHEADER as usual.
      NOTE: if you want chunked transfer too, you need to combine these two
      since you can only set one list of headers with CURLOPT_HTTPHEADER. */ 

    /* A less good option would be to enforce HTTP 1.0, but that might also
       have other implications. */ 
    {
      struct curl_slist *chunk = NULL;

      chunk = curl_slist_append(chunk, "Expect:");
      res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
      /* use curl_slist_free_all() after the *perform() call to free this
         list again */ 
    }
#endif

    /* Perform the request, res will get the return code */ 
    res = curl_easy_perform(curl);

    /* always cleanup */ 
    curl_easy_cleanup(curl);
  }
  return 0;
}

Solution 2

You can use CURLOPT_POSTFIELDS:

CURL *curl = curl_easy_init();

curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/api/endpoint");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "{\"hi\" : \"there\"}");

curl_easy_perform(curl);

Since CURLOPT_POSTFIELDS does not modify the payload in any way, it is very convenient for POSTing JSON data. Also note that, when CURLOPT_POSTFIELDS is supplied, it automatically enables CURLOPT_POST so there is no need to provide CURLOPT_POST in the request.

Solution 3

What about the required Content-Type header to match application/json just like the op is asking?

Using the CURLOPT_POSTFIELDS from two answers above as well as CURLOPT_POST, the Content-Type automatically gets set to application/x-www-form-urlencoded.

The only way for me to get the headers set correctly was to add what is outlined in this answer: JSON requests in C using libcurl

Solution 4

Also, you may use RAW input instead of adding extra backslashes:

curl_easy_setopt(curl, CURLOPT_POSTFIELDS, R"anydelim( {"hi" : "there"} )anydelim");

with delimiter or without it.

View more solutions

Comments

[Alastair Maw]

No worries. BTW, if you're writing C++ you should check out curlpp, which is a wrapper for straight C libcurl, and is a much nicer way of going about things: curlpp.org/index.php/examples/71-example-21

[dmitri]

Would you post a JSON array or a string in the same way? [1, "blah"] is not a form and there is no key-value pairs. You should probably mention that explicitly if this is the case.

[Raulp]

where to mention the API key for a server?