How can I get an HTTP response body as a string?

Question

I know there used to be a way to get it with Apache Commons as documented here:

http://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/HttpMethod.html

...and an example here:

http://www.kodejava.org/examples/416.html

...but I believe this is deprecated.

Is there any other way to make an http get request in Java and get the response body as a string and not a stream?

Solution 1

Every library I can think of returns a stream. You could use IOUtils.toString() from Apache Commons IO to read an InputStream into a String in one method call. E.g.:

URL url = new URL("http://www.example.com/");
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = con.getContentEncoding();
encoding = encoding == null ? "UTF-8" : encoding;
String body = IOUtils.toString(in, encoding);
System.out.println(body);

Update: I changed the example above to use the content encoding from the response if available. Otherwise it'll default to UTF-8 as a best guess, instead of using the local system default.

Solution 2

Here are two examples from my working project.

1. Using EntityUtils and HttpEntity

   HttpResponse response = httpClient.execute(new HttpGet(URL));
   HttpEntity entity = response.getEntity();
   String responseString = EntityUtils.toString(entity, "UTF-8");
   System.out.println(responseString);
   

2. Using BasicResponseHandler

   HttpResponse response = httpClient.execute(new HttpGet(URL));
   String responseString = new BasicResponseHandler().handleResponse(response);
   System.out.println(responseString);
   

Solution 3

Here's an example from another simple project I was working on using the httpclient library from Apache:

String response = new String();
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
nameValuePairs.add(new BasicNameValuePair("j", request));
HttpEntity requestEntity = new UrlEncodedFormEntity(nameValuePairs);

HttpPost httpPost = new HttpPost(mURI);
httpPost.setEntity(requestEntity);
HttpResponse httpResponse = mHttpClient.execute(httpPost);
HttpEntity responseEntity = httpResponse.getEntity();
if(responseEntity!=null) {
    response = EntityUtils.toString(responseEntity);
}

just use EntityUtils to grab the response body as a String. very simple.

Solution 4

This is relatively simple in the specific case, but quite tricky in the general case.

HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("http://stackoverflow.com/");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.getContentMimeType(entity));
System.out.println(EntityUtils.getContentCharSet(entity));

The answer depends on the Content-Type HTTP response header.

This header contains information about the payload and might define the encoding of textual data. Even if you assume text types, you may need to inspect the content itself in order to determine the correct character encoding. E.g. see the HTML 4 spec for details on how to do that for that particular format.

Once the encoding is known, an InputStreamReader can be used to decode the data.

This answer depends on the server doing the right thing - if you want to handle cases where the response headers don't match the document, or the document declarations don't match the encoding used, that's another kettle of fish.

Solution 5

Below is a simple way of accessing the response as a String using Apache HTTP Client library.

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;

//... 

HttpGet get;
HttpClient httpClient;

// initialize variables above

ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = httpClient.execute(get, responseHandler);

View more solutions

Comments

[eis]

Since the question and all the answers seem to be about apache libraries, this should be tagged as such. I don't see anything without using 3rdparty libs.

[tkruse]

related: stackoverflow.com/questions/21574478

[McDowell]

this will corrupt text in many cases as the method uses the system default text encoding which varies based on OS and user settings.

[WhiteFang34]

@McDowell: oops thanks, I linked the javadoc for the method with encoding but I forgot to use it in the example. I added UTF-8 to the example for now, although technically should use the Content-Encoding header from the response if available.

[Spidey]

Great usage of IOUtils. Nice pratical approach.

[Timur Yusupov]

Actually charset is specified in contentType like "charset=...", but not in contentEncoding, which contains something like 'gzip'

[Admin]

Do I have to close the InputStream?

[Tirtha]

The only problem I faced with method 1 is, the entity object is consumed when you do response.getEntity() and it is now available as responseString. if you try to do a response.getEntity() again, it'll return IllegalStateException.

[amIT]

this function causes the input stream to be closed, is there a way @WhiteFang34 i can print my response and continue to use the http entity

[Jaroslav Štreit]

Worked in my case - getting body from CloseableHttpClient response.

[Andreas L.]

What is httpClient?!

[spideringweb]

@AndreasL. httpClient is of type HttpClient ( org.apache.commons.httpclient package)

[SlimenTN]

This is a very good response, this is what I need to display, the response after posting data to the server. Well done.

[user1735921]

how to get it as HashMap ? I get response as Json, how to read that?

[Claus Ibsen]

Its so common to get the response content as string or byte array or something. Would be nice with an API directly on Entity to give you that. Having to look for this to find this util class.

[granadaCoder]

I upvoted. Please consider adding the full "imports" statements.

[Jack Leow]

Worth noting that the first method will get you the response body regardless of the HTTP status, whereas the second method throws exception for non-2xx HTTP statuses.