Apache HttpClient in Java, instream.toString = org.apache.http.conn.EofSensorInputStream

Question

I am GETting a page with Apache HttpClient and I want to store the server reply's http body into a string so I can then manipulate this string and print it to the console.

Unfortunately when running this method I get this message back:

17:52:01,862  INFO Driver:53 - fetchPage STARTING
17:52:07,580  INFO Driver:73 - fetchPage ENDING, took 5716
org.apache.http.conn.EofSensorInputStream@5e0eb724

The fetchPage Class:

public String fetchPage(String part){
    log.info("fetchPage STARTING");
    long start = System.currentTimeMillis();

    String reply;

    String searchurl = URL + URL_SEARCH_BASE + part + URL_SEARCH_TAIL;

    HttpClient httpclient = new DefaultHttpClient();
    HttpGet httpget = new HttpGet(searchurl);
    HttpResponse response;
    try {
        response = httpclient.execute(httpget);
        HttpEntity entity = response.getEntity();
        if (entity != null) {
            InputStream instream = entity.getContent();
            int l;
            byte[] tmp = new byte[2048];
            while ((l = instream.read(tmp)) != -1) {
            }
            long elapsedTimeMillis = System.currentTimeMillis()-start;
            log.info("fetchPage ENDING, took " + elapsedTimeMillis);
            reply = instream.toString();
            System.out.println(reply);
            return reply;
        }
    } catch (ClientProtocolException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    return null;
}

Solution

You are calling the toString on the InputStream after it has already read through. You need to create your string from the byte arrays. The simpler way to get the String version of the content is to use the EntityUtils.toString(HttpEntity)

The exact implementation would look like:

import org.apache.http.util.EntityUtils;

public String fetchPage(String part){
    log.info("fetchPage STARTING");
    long start = System.currentTimeMillis();

    String reply;

    String searchurl = URL + URL_SEARCH_BASE + part + URL_SEARCH_TAIL;

    HttpClient httpclient = new DefaultHttpClient();
    HttpGet httpget = new HttpGet(searchurl);
    HttpResponse response;
    try {
        response = httpclient.execute(httpget);
        HttpEntity entity = response.getEntity();
        if (entity != null) {
            return EntityUtils.toString(entity);
        }
    } catch (ClientProtocolException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    return null;
}

Comments

[Dominik]

If I add the long elapsedTimeMillis = System.currentTimeMillis()-start; right before return EntityUtils.toString(entity); the elapsed time for the procedure is about 1300ms, if I stop the time after String result = EntityUtils.toString(entity); and then return result it takes about 5500ms. Do you know why that is?

[ColinD]

@Jack: Probably because the client doesn't have to have received all the data the server is sending at the point where you get the entity. Creating the string requires actually reading all the content the server is sending.

[Joshua]

@Jack Murphy: the EntityUtils.toString(entity) is reading the stream. The client hasn't received all the data yet. You will want the latter to capture the amount of time of transfer.