How to get httpclient response as stream

Question

I am using httpclient 4.5.5 i want to get large files upto 1 gb in response. But it seems like

CloseableHttpResponse response = httpClient.execute(httpGet); HttpEntity entity = response.getEntity();

This returns whole response so its not good to have whole response in memory. Is there any way to get response as stream?

Solution 1

Apache HttpClient as of version 4.0 (as well as Apache HttpAsyncClient) supports full content streaming for both incoming and outgoing HTTP messages. Use HttpEntity to get access to the underling content input stream

CloseableHttpClient client = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://myhost/tons-of-stuff");
try (CloseableHttpResponse response1 = client.execute(httpGet)) {
    final HttpEntity entity = response1.getEntity();
    if (entity != null) {
        try (InputStream inputStream = entity.getContent()) {
            // do something useful with the stream
        }
    }
}

Solution 2

You need Apache Async Client.

HttpAsyncClient is the ASYNC version of Apache HttpClient. Apache HttpClient construct the whole response in memory, while with HttpAsyncClient, you can define a Handler (Consumer) to process the response while receiving data.

https://hc.apache.org/httpcomponents-asyncclient-4.1.x/index.html

Here is an example from their official example code

package org.apache.http.examples.nio.client;

import java.io.IOException;
import java.nio.CharBuffer;
import java.util.concurrent.Future;

import org.apache.http.HttpResponse;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.nio.IOControl;
import org.apache.http.nio.client.methods.AsyncCharConsumer;
import org.apache.http.nio.client.methods.HttpAsyncMethods;
import org.apache.http.protocol.HttpContext;

/**
 * This example demonstrates an asynchronous HTTP request / response exchange with
 * a full content streaming.
 */
public class AsyncClientHttpExchangeStreaming {

    public static void main(final String[] args) throws Exception {
        CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault();
        try {
            httpclient.start();
            Future<Boolean> future = httpclient.execute(
                    HttpAsyncMethods.createGet("http://httpbin.org/"),
                    new MyResponseConsumer(), null);
            Boolean result = future.get();
            if (result != null && result.booleanValue()) {
                System.out.println("Request successfully executed");
            } else {
                System.out.println("Request failed");
            }
            System.out.println("Shutting down");
        } finally {
            httpclient.close();
        }
        System.out.println("Done");
    }

    static class MyResponseConsumer extends AsyncCharConsumer<Boolean> {

        @Override
        protected void onResponseReceived(final HttpResponse response) {
        }

        @Override
        protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
            while (buf.hasRemaining()) {
                System.out.print(buf.get());
            }
        }

        @Override
        protected void releaseResources() {
        }

        @Override
        protected Boolean buildResult(final HttpContext context) {
            return Boolean.TRUE;
        }

    }

}

Comments

[Robert]

Just posting links to overview pages is not an answer. External links break very fast and SO answers remain then without value for readers. Therefore you should include the relevant (code) part into your answer!