How to use HttpClientBuilder with Http proxy?

Question

I am trying to set proxy for a request I am making using HttpClientBuilder as follows:

        CredentialsProvider credsProvider = new BasicCredentialsProvider();
        UsernamePasswordCredentials usernamePasswordCredentials = new UsernamePasswordCredentials(proxyUser, proxyPassword);
        credsProvider.setCredentials(new AuthScope(proxyHost, proxyPort), usernamePasswordCredentials);

        builder.useSystemProperties();
        builder.setProxy(new HttpHost(proxyHost, proxyPort));
        builder.setDefaultCredentialsProvider(credsProvider);
        builder.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());

where builder is:

    HttpClientBuilder builder = HttpClientBuilder.create();

However, I get this exception when I execute this request:

java.lang.RuntimeException: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported
Caused by: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:108) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.1.jar:4.5.1]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.1.jar:4.5.1]

(exception shortened for brevity)

Since this is an HTTP proxy, I don't want to change the scheme to HTTPS, which anyways won't work. How do I get this working?

Solution 1

java.lang.RuntimeException: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported

Why this problem occurs?

Ans: This actually happens because you forget to register a connection socket factory for the 'http' scheme.

Plain 'http' scheme must be used to establish an intermediate connection to the proxy itself before 'https' tunneling could be employed.

---

For operational purpose, you can try this code:

CloseableHttpClient client = HttpClients.custom()
           .setRoutePlanner(new
 SystemDefaultRoutePlanner(ProxySelector.getDefault()))
           .build();

---

I would also suggest simple code for your research. Hope it can save you.

ClientExecuteProxy.java

package org.apache.http.examples.client;

import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

/**
 * How to send a request via proxy.
 *
 * @since 4.0
 */
public class ClientExecuteProxy {

    public static void main(String[] args)throws Exception {
        CloseableHttpClient httpclient = HttpClients.createDefault();
        try {
            HttpHost target = new HttpHost("httpbin.org", 443, "https");
            HttpHost proxy = new HttpHost("127.0.0.1", 8080, "http");

            RequestConfig config = RequestConfig.custom()
                    .setProxy(proxy)
                    .build();
            HttpGet request = new HttpGet("/");
            request.setConfig(config);

            System.out.println("Executing request " + request.getRequestLine() + " to " + target + " via " + proxy);

            CloseableHttpResponse response = httpclient.execute(target, request);
            try {
                System.out.println("----------------------------------------");
                System.out.println(response.getStatusLine());
                System.out.println(EntityUtils.toString(response.getEntity()));
            } finally {
                response.close();
            }
        } finally {
            httpclient.close();
        }
    }

}

Are you using using CloudantClient java API for Cloudant DB?

Ans:

If YES, then It turned out the issue with HTTP when setting a proxy was a bug at our end (sorry about that). We released 1.2.1 with the fix for this problem. You can download jar file from here. (Collected from mike-rhodes's answer)

---

UPDATE

How do I specify the credentials for the proxy here?

From HTTP authentication,

By default, httpclient will not provide credentials preemptively, it will first create a HTTP request without authentication parameters. This is by design, as a security precaution, and as part of the spec. But, this causes issues if you don't retry the connection, or wherever you're connecting to expects you to send authentication details on the first connection. It also causes extra latency to a request, as you need to make multiple calls, and causes 401s to appear in the logs.

The workaround is to use an authentication cache to pretend that you've already connected to the server once. This means you'll only make one HTTP call.

CloseableHttpClient httpclient = HttpClientBuilder.create().build();

HttpHost targetHost = new HttpHost("localhost", 80, "http");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
        new AuthScope(targetHost.getHostName(), targetHost.getPort()),
        new UsernamePasswordCredentials("username", "password"));

// Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// Generate BASIC scheme object and add it to the local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put(targetHost, basicAuth);

// Add AuthCache to the execution context
HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache);

HttpGet httpget = new HttpGet("/");
for (int i = 0; i < 3; i++) {
    CloseableHttpResponse response = httpclient.execute(
            targetHost, httpget, context);
    try {
        HttpEntity entity = response.getEntity();

    } finally {
        response.close();
    }
}

N.B: You need to trust the host you're connecting to, and if you're using HTTP, your username and password will be sent in cleartext (well, base64, but that doesn't count).

You should also be using a much more specific Authscope rather than relying on AuthScope.ANY_HOST and AuthScope.ANY_PORT like in your example.

Credit goes to Cetra

Related Links:

1. HttpClientBuilder basic auth
2. Apache HttpClient 4.1 - Proxy Authentication

Solution 2

What you have should be very close to working. I would make the following simple changes:

builder.useSystemProperties();

Delete the call to useSystemProperties. It isn't documented well, but when you set the Proxy (as you do in the next line), it overrides this, so just remove that line.

builder.setProxy(new HttpHost(proxyHost, proxyPort));

Call the HttpHost constructor with the explicit 'scheme' parameter. This is where you are getting the error, so make it explicit:

String proxyScheme = "http";
builder.setProxy(new HttpHost(proxyHost, proxyPort, proxyScheme));

Note: you did not say, but based on the usage of "BasicCredentialsProvider", this is only giving you "Basic" authentication. Basic is only encoded and is not really secure. For Digest or NTLM or Kerberos you will need different code.

Solution 3

I think the problem is with your HttpClient, not the proxy. Did you try to create your HttpClient by using HttpClientBuilder.build()

HttpClient client = builder.build();

Comments

[An Do]

Which version of httpcore you are using?

[jobin]

HttpComponents Version 4.5.1

[jobin]

How do I specify the credentials for the proxy here?

[SkyWalker]

@i08in I have updated my answer about credentials for the proxy and given some useful links. Hope it will clarify you.

[Adrien]

just a minor note, but when using a proxy, the auth challenges come back with 407 code, rather than 401 (used by servers)

[Adrien]

the proxy will advertise supported auth schemes in its initial challenge, if you're using Basic, and the proxy didn't advertise it, it may just block you.

[SkyWalker]

@Adrien, yes it is. 401 status code indicates that before a resource can be accessed, the client must be authorized by the server. 407 status code is very similar to the 401 status code, and means that the client must be authorized by the proxy before the request can proceed.

[Adrien]

sure, but you won't see a 401 for proxy auth unless the proxy intercepted the connection. You'll see a 401 for server auth which may have come back via a proxy which you either authed to or not.