How to get the HTTP response string using Curl in C++

Question

I'm very new to HTTP commands and the libcurl library. I know how to get the HTTP response code but not the HTTP response string. Following is the code snippet that I wrote to get the response code. Any help on how to get the response string will be highly appreciated!!!

curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
CURLcode ret = curl_easy_perform(curl);

if (ret != CURLE_OK) {
    LOG(INFO) << "Failed to perform the request. "
              << "Return code: " << ret;
    return false;
}

std::unique_ptr<int64_t> httpCode(new int64_t);
// Get the last response code.
ret = curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, httpCode.get());
if (ret != CURLE_OK) {
  LOG(INFO) << "curl_easy_getinfo failed to retrieve http code. "
            << "Return code: " << ret;
  return false;
}

I tried doing this as well to get the HTTP response string in readBuffer.

static size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp)
{
    ((std::string*)userp)->append((char*)contents, size * nmemb);
    return size * nmemb;
}

std::string readBuffer;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);

CURLcode ret = curl_easy_perform(curl);
cout << readBuffer << "\n";

But the readBuffer is empty. I don't understand where I am going wrong. Any pointers on how to solve this will be really nice!

Solution 1

There doesn't look to be much wrong with your code. I ran the following code, based on yours, to read the front page of the BBC news website:

#include <iostream>
#include <string>
#include <curl/curl.h>

size_t WriteCallback(char *contents, size_t size, size_t nmemb, void *userp)
{
    ((std::string*)userp)->append((char*)contents, size * nmemb);
    return size * nmemb;
}

int main(int argc, char * argv[])
{
    curl_global_init(CURL_GLOBAL_ALL);

    CURL* easyhandle = curl_easy_init();
    std::string readBuffer;

    curl_easy_setopt(easyhandle, CURLOPT_URL, "http://www.bbc.co.uk/news");
    curl_easy_setopt(easyhandle, CURLOPT_VERBOSE, 1L);
    curl_easy_setopt(easyhandle, CURLOPT_PROXY, "http://my.proxy.net");   // replace with your actual proxy
    curl_easy_setopt(easyhandle, CURLOPT_PROXYPORT, 8080L);
    curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, WriteCallback);
    curl_easy_setopt(easyhandle, CURLOPT_WRITEDATA, &readBuffer);

    curl_easy_perform(easyhandle);

    std::cout << readBuffer << std::endl;

    return 0;
}

... and I got the full HTML response. NB I had to use a proxy, and I enabled verbose mode to see what was happening. Also NB; the HTTP server might be fussy about the URL you give it: if I replace http://www.bbc.co.uk/news with http://www.bbc.co.uk/news/ (i.e. with a trailing slash) then I get no data back (a response length of zero), as noted by the verbose curl output:

Host: www.bbc.co.uk
Accept: */*
Proxy-Connection: Keep-Alive

< HTTP/1.1 301 Moved Permanently
< X-Cache-Action: PASS (no-cache-control)
< Vary: Accept-Encoding
< X-Cache-Age: 0
< Content-Type: text/html;charset=utf-8
< Date: Mon, 10 Jul 2017 16:42:20 GMT
< Location: http://www.bbc.co.uk/news
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< Content-Length: 0
< X-Cache: MISS from barracuda1.[my proxy].net
< Connection: keep-alive
< 
* Connection #0 to host [my proxy] left intact

Here, Content-Length is 0 and the WriteCallback() function is never called. Hope this helps.

Solution 2

For the numerical response code, getinfo with CURLINFO_RESPONSE_CODE is the way to go:

long response_code;
curl_easy_getinfo(handle, CURLINFO_RESPONSE_CODE,&response_code);

However there is no equivalent getinfo capture for the server's response text. If you need the server's text, inspect the raw HTTP headers. There are two ways to do this:

1. Enable writing headers to the payload with CURLOPT_HEADER, then extract the headers from the combined payload, splitting the body on \n\n

2. Set a header callback function with CURLOPT_HEADERFUNCTION and parse directly from that