Using Apache httpclient for https

Question

I have enabled https in tomcat and have a self-signed certificate for server auth. I have created an http client using Apache httpClient. I have set a trust manager loading the server certificate. The http client can connect with server no problem. To see what is going on I enabled debugging:

System.setProperty("javax.net.debug", "ssl");

I saw the following which I can not understand at all:

***
adding as trusted cert:
  Subject: CN=Me, OU=MyHouse, O=Home, L=X, ST=X, C=BB
  Issuer:  CN=Me, OU=MyHouse, O=Home, L=X, ST=X, C=BB
  Algorithm: RSA; Serial number: 0x4d72356b
  Valid from Sat Mar 05 15:06:51 EET 2011 until Fri Jun 03 16:06:51 EEST 2011 

My certificate is displayed and is added to truststore (as I see). Then:

trigger seeding of SecureRandom
done seeding SecureRandom

Here is the part from debugging traces I do not get:

trustStore is: C:\Program Files\Java\jre6\lib\security\cacerts
trustStore type is : jks
trustStore provider is : 
init truststore
adding as trusted cert:
  Subject: CN=SwissSign Platinum CA - G2, O=SwissSign AG, C=CH
  Issuer:  CN=SwissSign Platinum CA - G2, O=SwissSign AG, C=CH
  Algorithm: RSA; Serial number: 0x4eb200670c035d4f
  Valid from Wed Oct 25 11:36:00 EEST 2006 until Sat Oct 25 11:36:00 EEST 2036

adding as trusted cert:
  Subject: [email protected], CN=http://www.valicert.com/, OU=ValiCert Class 1 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network
  Issuer:  [email protected], CN=http://www.valicert.com/, OU=ValiCert Class 1 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network
  Algorithm: RSA; Serial number: 0x1
  Valid from Sat Jun 26 01:23:48 EEST 1999 until Wed Jun 26 01:23:48 EEST 2019

It seems that it also uses the default java trust store! My question is why does this happen?

In my code I specify explicitly a specific trust-store to use (via truststoremanagers). I was expecting only this to be used. It seems that both my truststore and java's default is being used. Is this how it is supposed to work?

UPDATE:
I tried the following:

System.out.println("TMF No:"+tmf.getTrustManagers().length);
System.out.println("Class is "+tmf.getTrustManagers()[0].getClass().getName());  

I thought that I should see 2 trust managers, since 2 keystores (mine and java's default appear to be used).
But the result was only 1 trust manager!

TMF No:1
Class is com.sun.net.ssl.internal.ssl.X509TrustManagerImpl  

UPDATE2: As you see in the code bellow I specify my keystore.My expectation is that only this should be used (not this and cacert as well)

    HttpClient client = new DefaultHttpClient();
    SSLContext sslContext = SSLContext.getInstance("TLS");      

    TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    KeyStore ks = KeyStore.getInstance("JKS");
    File trustFile = new File("clientTrustStore.jks");
    ks.load(new FileInputStream(trustFile), null);
    tmf.init(ks);
    sslContext.init(null, tmf.getTrustManagers(),null);  
    SSLSocketFactory sf = new SSLSocketFactory(sslContext); 
    sf.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
    Scheme scheme = new Scheme("https", sf, 443);
    client.getConnectionManager().getSchemeRegistry().register(scheme);
    httpGet = new HttpGet("https://localhost:8443/myApp");
    HttpResponse httpResponse = client.execute(httpGet);

Does not make sense to me.

Solution 1

I put together this test app to reproduce the issue using the HTTP testing framework from the Apache HttpClient package:

ClassLoader cl = HCTest.class.getClassLoader();
URL url = cl.getResource("test.keystore");
KeyStore keystore  = KeyStore.getInstance("jks");
char[] pwd = "nopassword".toCharArray();
keystore.load(url.openStream(), pwd);

TrustManagerFactory tmf = TrustManagerFactory.getInstance(
        TrustManagerFactory.getDefaultAlgorithm());
tmf.init(keystore);
TrustManager[] tm = tmf.getTrustManagers();

KeyManagerFactory kmfactory = KeyManagerFactory.getInstance(
        KeyManagerFactory.getDefaultAlgorithm());
kmfactory.init(keystore, pwd);
KeyManager[] km = kmfactory.getKeyManagers();

SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(km, tm, null);

LocalTestServer localServer = new LocalTestServer(sslcontext);
localServer.registerDefaultHandlers();

localServer.start();
try {

    DefaultHttpClient httpclient = new DefaultHttpClient();
    TrustStrategy trustStrategy = new TrustStrategy() {

        public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
            for (X509Certificate cert: chain) {
                System.err.println(cert);
            }
            return false;
        }

    };

    SSLSocketFactory sslsf = new SSLSocketFactory("TLS", null, null, keystore, null,
            trustStrategy, new AllowAllHostnameVerifier());
    Scheme https = new Scheme("https", 443, sslsf);
    httpclient.getConnectionManager().getSchemeRegistry().register(https);

    InetSocketAddress address = localServer.getServiceAddress();
    HttpHost target1 = new HttpHost(address.getHostName(), address.getPort(), "https");
    HttpGet httpget1 = new HttpGet("/random/100");
    HttpResponse response1 = httpclient.execute(target1, httpget1);
    System.err.println(response1.getStatusLine());
    HttpEntity entity1 = response1.getEntity();
    EntityUtils.consume(entity1);
    HttpHost target2 = new HttpHost("www.verisign.com", 443, "https");
    HttpGet httpget2 = new HttpGet("/");
    HttpResponse response2 = httpclient.execute(target2, httpget2);
    System.err.println(response2.getStatusLine());
    HttpEntity entity2 = response2.getEntity();
    EntityUtils.consume(entity2);
} finally {
    localServer.stop();
}

Even though, Sun's JSSE implementation appears to always read the trust material from the default trust store for some reason, it does not seem to get added to the SSL context and to impact the process of trust verification during the SSL handshake.

Here's the output of the test app. As you can see, the first request succeeds whereas the second fails as the connection to www.verisign.com is rejected as untrusted.

[
[
  Version: V1
  Subject: CN=Simple Test Http Server, OU=Jakarta HttpClient Project, O=Apache Software Foundation, L=Unknown, ST=Unknown, C=Unknown
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3

  Key:  Sun DSA Public Key
    Parameters:DSA
    p:     fd7f5381 1d751229 52df4a9c 2eece4e7 f611b752 3cef4400 c31e3f80 b6512669
    455d4022 51fb593d 8d58fabf c5f5ba30 f6cb9b55 6cd7813b 801d346f f26660b7
    6b9950a5 a49f9fe8 047b1022 c24fbba9 d7feb7c6 1bf83b57 e7c6a8a6 150f04fb
    83f6d3c5 1ec30235 54135a16 9132f675 f3ae2b61 d72aeff2 2203199d d14801c7
    q:     9760508f 15230bcc b292b982 a2eb840b f0581cf5
    g:     f7e1a085 d69b3dde cbbcab5c 36b857b9 7994afbb fa3aea82 f9574c0b 3d078267
    5159578e bad4594f e6710710 8180b449 167123e8 4c281613 b7cf0932 8cc8a6e1
    3c167a8b 547c8d28 e0a3ae1e 2bb3a675 916ea37f 0bfa2135 62f1fb62 7a01243b
    cca4f1be a8519089 a883dfe1 5ae59f06 928b665e 807b5525 64014c3b fecf492a

  y:
    f0cc639f 702fd3b1 03fa8fa6 676c3756 ea505448 23cd1147 fdfa2d7f 662f7c59
    a02ddc1a fd76673e 25210344 cebbc0e7 6250fff1 a814a59f 30ff5c7e c4f186d8
    f0fd346c 29ea270d b054c040 c74a9fc0 55a7020f eacf9f66 a0d86d04 4f4d23de
    7f1d681f 45c4c674 5762b71b 808ded17 05b74baf 8de3c4ab 2ef662e3 053af09e

  Validity: [From: Sat Dec 11 14:48:35 CET 2004,
               To: Tue Dec 09 14:48:35 CET 2014]
  Issuer: CN=Simple Test Http Server, OU=Jakarta HttpClient Project, O=Apache Software Foundation, L=Unknown, ST=Unknown, C=Unknown
  SerialNumber: [    41bafab3]

]
  Algorithm: [SHA1withDSA]
  Signature:
0000: 30 2D 02 15 00 85 BE 6B   D0 91 EF 34 72 05 FF 1A  0-.....k...4r...
0010: DB F6 DE BF 92 53 9B 14   27 02 14 37 8D E8 CB AC  .....S..'..7....
0020: 4E 6C 93 F2 1F 7D 20 A1   2D 6F 80 5F 58 AE 33     Nl.... .-o._X.3

]
HTTP/1.1 200 OK
[
[
  Version: V3
  Subject: CN=www.verisign.com, OU=" Production Security Services", O="VeriSign, Inc.", STREET=487 East Middlefield Road, L=Mountain View, ST=California, OID.2.5.4.17=94043, C=US, SERIALNUMBER=2497886, OID.2.5.4.15="V1.0, Clause 5.(b)", OID.1.3.6.1.4.1.311.60.2.1.2=Delaware, OID.1.3.6.1.4.1.311.60.2.1.3=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 20699622354183393041832954221256409980425015218949582822286196083815087464214375375678538878841956356687753084333860738385445545061253653910861690581771234068858443439641948884498053425403458465980515883570440998475638309355278206558031134532548167239684215445939526428677429035048018486881592078320341210422026566944903775926801017506416629554190534665876551381066249522794321313235316733139718653035476771717662585319643139144923795822646805045585537550376512087897918635167815735560529881178122744633480557211052246428978388768010050150525266771462988042507883304193993556759733514505590387262811565107773578140271
  public exponent: 65537
  Validity: [From: Wed May 26 02:00:00 CEST 2010,
               To: Sat May 26 01:59:59 CEST 2012]
  Issuer: CN=VeriSign Class 3 Extended Validation SSL SGC CA, OU=Terms of use at https://www.verisign.com/rpa (c)06, OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US
  SerialNumber: [    53d2bef9 24a7245e 83ca01e4 6caa2477]

Certificate Extensions: 10
[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [accessMethod: 1.3.6.1.5.5.7.48.1
   accessLocation: URIName: http://EVIntl-ocsp.verisign.com, accessMethod: 1.3.6.1.5.5.7.48.2
   accessLocation: URIName: http://EVIntl-aia.verisign.com/EVIntl2006.cer]
]

...

]
Exception in thread "main" javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
    at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:345)
    at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
    at org.apache.http.conn.ssl.SSLSocketFactory.createLayeredSocket(SSLSocketFactory.java:446)
...

Solution 2

When I used Apache HTTP Client 4.3, I was using the Pooled or Basic Connection Managers to the HTTP Client. I noticed, from using java SSL debugging, that these classes loaded the cacerts trust store and not the one I had specified programmatically.

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
BasicHttpClientConnectionManager cm = new BasicHttpClientConnectionManager();
builder.setConnectionManager( cm );

I wanted to use them but ended up removing them and creating an HTTP Client without them. Note that builder is an HttpClientBuilder.

I confirmed when running my program with the Java SSL debug flags, and stopped in the debugger. I used -Djavax.net.debug=ssl as a VM argument. I stopped my code in the debugger and when either of the above *ClientConnectionManager were constructed, the cacerts file would be loaded.

Solution 3

This is what worked for me:

    KeyStore keyStore  = KeyStore.getInstance("PKCS12");
    FileInputStream instream = new FileInputStream(new File("client-p12-keystore.p12"));
    try {
        keyStore.load(instream, "password".toCharArray());
    } finally {
        instream.close();
    }

    // Trust own CA and all self-signed certs
    SSLContext sslcontext = SSLContexts.custom()
        .loadKeyMaterial(keyStore, "password".toCharArray())
        //.loadTrustMaterial(trustStore, new TrustSelfSignedStrategy())
        .build();
    // Allow TLSv1 protocol only
    SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
        sslcontext,
        new String[] { "TLSv1" },
        null,
        SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); //TODO
    CloseableHttpClient httpclient = HttpClients.custom()
        .setHostnameVerifier(SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER) //TODO
        .setSSLSocketFactory(sslsf)
        .build();
    try {

        HttpGet httpget = new HttpGet("https://localhost:8443/secure/index");

        System.out.println("executing request" + httpget.getRequestLine());

        CloseableHttpResponse response = httpclient.execute(httpget);
        try {
            HttpEntity entity = response.getEntity();

            System.out.println("----------------------------------------");
            System.out.println(response.getStatusLine());
            if (entity != null) {
                System.out.println("Response content length: " + entity.getContentLength());
            }
            EntityUtils.consume(entity);
        } finally {
            response.close();
        }
    } finally {
        httpclient.close();
    }
}

This code is a modified version of http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/examples/org/apache/http/examples/client/ClientCustomSSL.java

Solution 4

According to the documentation you need to specify the key store:

Protocol authhttps = new Protocol("https",  
      new AuthSSLProtocolSocketFactory(
          new URL("file:my.keystore"), "mypassword",
          new URL("file:my.truststore"), "mypassword"), 443); 

 HttpClient client = new HttpClient();
 client.getHostConfiguration().setHost("localhost", 443, authhttps);

View more solutions

Comments

[Cratylus]

I am not using commons.I am using hc.apache.org/httpcomponents-client-ga. I am doing something similar Scheme scheme = new Scheme("https", 443, sf); client.getConnectionManager().getSchemeRegistry().regi‌​ster(scheme); and sf is SSLSocketFactory sf = new SSLSocketFactory(sslContext, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); and in sslContext I have set my trust store manager.I can update the post with some code if it will help

[extraneon]

@user384706 I'm getting a bit confused. The problem is that the client is using 2 TrustManagers (which SSLContext.init says it should not) or is it the tomcat server which behaves strangely?

[Cratylus]

No problem in Tomcat.Tomcat sends the certificate I have configured.Problem is in client part.I have specified a trusted trustore and was expecting only this to be used.The debuging info show that mine as well as the default java truststore is used.I have updated my post to show some code in client part

[Abhishek]

Hi Oleg, I am facing a similar issue but only when i am sending a POST request and not via GET. Any insight?

[pulkitsinghal]

So, is there a bug filed for this somewhere that we can track for a solution?

[pulkitsinghal]

I found out that my JVM did not have the JCE jar files necessary to decrypt the 4096 bits key, perhaps that the same issue with 2048 bits key that you're testing?

[Deepesh M]

@Cratylus, I am facing the same issue while testing httpclient 4.0 with self-signed certificates. Can you post the code how you got it working?

[Cratylus]

@DeepeshM:I did not have any working issues.My post was related to why certificates part of cacerts were printed.My concern was that both the cacerts and my truststore was used

[Deepesh M]

@Cratylus, oh I am having working issue. Even though I am adding my own truststore with TrustManagerFactory still it is not adding to the default trust. When I set the debug, it does not show my certificates being added. I have posted the issue at stackoverflow.com/questions/15164831/….

[Ben Green]

Thank you so much for this. I couldn't for the life of me work out what the problem was, and then after reading this noticed that the builder basically ignores the sslContext you provide if you have passed in a connection manager.

[Admin]

How we can allow connections (insecure) to SSL sites without certs using HTTP client ?

[Admin]

How we can allow connections to SSL sites without certs insecure way in above code?

[Admin]

Which version of HTTPClient you used ?

[EpicPandaForce]

@user4567570 It was 4.3.5 I think. To connect insecurely, you need to allow the Hostname Verifier to allow all, but you also need to specify to use a strategy that trusts everyone: sslContextBuilder.loadTrustMaterial(new TrustAllStrategy()); and TrustAllStrategy implements TrustStrategy, and all it does is return true;

[Admin]

Hey, could you please guide on this stackoverflow.com/questions/34655031/…?

[Narendra Singh]

how to get keystore and password?

[EpicPandaForce]

KeyStore keyStore = KeyStore.getInstance("PKCS12", BouncyCastleProvider.PROVIDER_NAME); byteArrayInputStream = new ByteArrayInputStream(keyStoreBytes); keyStore.load(byteArrayInputStream, keyStorePassword); i generated that thing with bouncycastle

[lorrainebatol]

This answer deserves more love. I have a task for SSL and conn pooling. I could swear I ran my manual tests and integration tests successfully right before my eyes for SSL. Then I did some code cleanup and nonchalantly inserted pooling conn manager. You could imagine my distress when everything started failing! My fix is instead of setting the pool values as conn manager,can actually set it directly on httpClient without instantiating and setting through setConnectionManager.