Using Java To Download Files From a HTTPS URL

Question

I have written some code to download a file from a website. The code works fine against a test http url. As soon as I then change the URL to the https, I get a connect time out.

System.setProperty("http.proxyHost","trproxy.rwe.com") ;
System.setProperty("http.proxyPort", "80") ;
Authenticator.setDefault (new MyAuthenticator("USER","PW"));
//URL url = new URL("http","www.treasury.gov",80,"/ofac/downloads/sdn.csv",new    sun.net.www.protocol.http.Handler());  THIS WORKS
URL url = new URL("https", "downloads.elexonportal.co.uk",443,"/bmradataarchive/download?key=MYKEY&filename="+filename,new sun.net.www.protocol.https.Handler());
url.openConnection();
InputStream reader = url.openStream();
FileOutputStream writer = new FileOutputStream("C:/downloads/"+filename);

If I copy the https url into a browser, I am asked where I wish to save the file and it works fine. Any help greatly appreciated. I have tried this but did not work

Thanks Chris

Solution 1

You may be having certificate issues. This is typically the problem I have encountered in the past when working with HTTPS connections in Java.

First, you should check and see if the URL to which you are attempting to connect has a signed certificate by a well-known trusted root CA, and is valid (not expired).

I would recommend opening the URL in your browser and checking the certificate information.

Just a FYI, there may be a disconnect between the Trusted Root CAs recognized by your browser and those recognized by Java. Here is another Stackoverflow question about how to get those recognized by Java: How can I get a list of trusted root certificates in Java?

If this is a self-signed certificate, then there are hoops you will need to jump through regarding importing it into and using a local Keystore. There are numerous sites and blogs that guide you through doing this, here is one such blog (not mine): Adding self-signed https certificates to java keystore

Also, while you are testing with the browser, this will help you verify that there are no proxy issues. You should definitely check your browser's settings to determine whether or not you are going through a proxy server.

You should definitely look into using HttpClient instead of java.net.URL. Here is the Apache page for HttpClient 4.2.1.

Finally, if you are looking to do a file transfer via HTTP or HTTPS, you may want to consider WebDAV.

I have used Jakarta Slide WebDAV Client for this in the past. Unfortunately, it looks like Slide is retired at this point, but there are alternatives you can find with a little bit of searching.

ADDITION

I copied down your source code sample and looked at it more closely. Looks like you set properties for http, but not https.

HTTPS has separate properties:

• https.proxyHost
• https.proxyPort

Try setting:

System.setProperty("https.proxyHost","trproxy.rwe.com") ; 
System.setProperty("https.proxyPort", "443") ; 

Look at section 2.2 on Oracle's Java Networking and Proxies.

http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html

Solution 2

Looks like your problem could be with the proxy. It should also support https. Also, you should use HttpClient 4 for downloading the files, its a mature library for making http connections.

Comments

[Chris]

Philip thanks for taking the time to reply, it was the System.setProperty("https... that got it. I had been looking so hard this solution, so simple!!