getting javax.net.ssl.SSLException: Received fatal alert: protocol_version while scraping data using Jsoup

Question

I am trying to get data from a site using Jsoup. Link to the site is Click here!

Here is my code to fetch the data. `

    // WARNING: do it only if security isn't important, otherwise you have 
    // to follow this advices: http://stackoverflow.com/a/7745706/1363265
    // Create a trust manager that does not validate certificate chains
    TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager(){
        public X509Certificate[] getAcceptedIssuers(){return null;}
        public void checkClientTrusted(X509Certificate[] certs, String authType){}
        public void checkServerTrusted(X509Certificate[] certs, String authType){}
    }};

    // Install the all-trusting trust manager
    try {
        SSLContext sc = SSLContext.getInstance("TLS");
        sc.init(null, trustAllCerts, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
    } catch (Exception e) {
        ;
    }`

String url = "https://www.sos.nh.gov/corporate/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=Starting&SearchStr="+query+"&SearchType=Search"; Connection.Response response = Jsoup.connect(url).timeout(30000) .method(Connection.Method.GET) .userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0")
.execute(); Document document = response.parse();

Please suggest me what is my mistake here.

Solution 1

You want to use Java 8 here since it supports TLSv1.2 by default with additional required cipher suites.

Why not Java 7?

I tested on my box with Java 7 (1.7.0_45) and got the same error.

I activated the debugging messages and forced TLSv1.2.

System.setProperty("javax.net.debug", "all");
System.setProperty("https.protocols", "TLSv1.2");

Then I hit this new error:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Finally, I went to Comodoca's SSL analyzer and see something interesting. According to SSL analyzer, the site you're targeting has only enabled the following cipher suites:

Cipher Suites Enabled
Name  (ID)                                       Key Size (in bits)
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256  (0xC02F)  128   ECDH 256-bit (P-256) 
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384  (0xC030)  256   ECDH 256-bit (P-256) 
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256  (0x9E)      128   DH 2048-bit  
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384  (0x9F)      256   DH 2048-bit

(see Full details)

On my side, I don't have any of the above suites. Check if you have them:

SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, null, new java.security.SecureRandom());

String[] scs = sc.getSocketFactory().getSupportedCipherSuites();
Arrays.sort(scs);

for(String s : scs) {
   System.out.println(s);
}

See the SSLSocketFactoryEx for enabling the needed cipher suites.

Why Java 8?

On the other hand, I succeed in runnnig the code by moving from Java 7 to Java 8 (1.8.0_20) which support TLS v1.2 by default and provides the needed cipher suites.

Here is a trimmed list of supported cipher suites (71 suites in total) for Java 8 (1.8.0_20) on Windows 7.

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
...
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Snippet

try {
    // Create a trust manager that does not validate certificate chains
    TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return null;
        }

        public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
        }

        public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
        }
    } };

    // Install the all-trusting trust manager
    SSLContext sc = SSLContext.getInstance("SSL");
    sc.init(null, trustAllCerts, new java.security.SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

    // Fetch url
    String url = "https://www.sos.nh.gov/corporate/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=All&SearchStr=facebook&SearchType=Search";

    Connection.Response response = Jsoup //
            .connect(url) //
            .timeout(60000) //
            .method(Connection.Method.GET) //
            .userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0") //
            .execute();

    Document document = response.parse();
    System.out.println(document);
} catch (Exception e) {
    e.printStackTrace();
}

Final thought:

When it comes to security, ALWAYS use the latest updated version.

Solution 2

(From comment for closure, expanded a little for future finders)

By experiment, that site requires protocol version TLSv1.2 and although Java7 JSSE implements this, client side by default disables 1.2 and 1.1. Java8 does enable them by default; or in Java7 since Jsoup uses HttpsURLConnection you can change the enabled versions with system property https.protocols. You need to include at least TLSv1.2 and for greatest flexibility should use all currently acceptable protocols https.protocols=TLSv1,TLSv1.1,TLSv1.2.

Also, using that all-trusting TrustManager means that pretty much any baddie with access to your network can fake this site and expose any sensitive data you send it. It's better to set your local truststore so it accepts the certificate(s) and thus server(s) you need, but not bogus ones.

Comments

[Marged]

Which ciphers does the site use ? Which Java version do you use ? It's likely the site demands for sslv3 and your Java removed support

[graphics123]

java version is 7. Which version of Java will support it? Or any third party library should do it?

[Marged]

You have to check which Ssl / tls version the said web server supports. It could be you are asking for more than it supports. Google how to use OpenSSL for getting this information, I am sure you will find info here on stackoverflow.

[dave_thompson_085]

@Marged + Subhasish: Other way: that site accepts TLSv1.2 only, and Java7 client by default doesn't do 1.2 (or 1.1). Since Jsoup uses HttpsURLConnection, if you can't upgrade to Java8, you can set system property https.protocols=TLSv1,TLSv1.1,TLSv1.2. Also, using that all-trusting TrustManager means that pretty much anyone with access to your network can fake this site and expose any sensitive data you send it.

[Marged]

@dave_Thompson_085 I was just guessing, good you checked the details. Want to turn this into an answer ?

[graphics123]

@dave_thompson Yeah I added the system property and it worked like a charm,. Please convert it to an answer , I would love to mark it as accepted . Thanks. Marged Thanks for the help from your side as well.

[Stephan]

See also: stackoverflow.com/q/33439478/363573

[graphics123]

Due to some reason , this doesn't work anymore with java 7

[graphics123]

Could you please add a small snippet to the answer, so that, I can accept it.

[Stephan]

@Subhasish I have added a snippet in the post.