HTTP Post to API return 403 FORBIDDEN

Question

I have to upload an XML file to an API. This is API is secured by a signed certificate, which I got from the issuer of the API.

Now, I got two use cases. First, I have to download some files from this API. This is working perfectly with the following code:

final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ip, port));

final URL url = new URL(linkToFile);
final HttpsURLConnection conn = (HttpsURLConnection) url.openConnection(proxy);
conn.setSSLSocketFactory(this.http.createSSLContext().getSocketFactory());
try (
            InputStream inputStream = zipUrlConn.getInputStream();
            ZipInputStream stream = new ZipInputStream(inputStream);) {

    // Do stuff with ZipInputStream here
}

The createSSLContext() method looks as follows:

public SSLContext createSSLContext() throws NoSuchAlgorithmException, CertificateException, IOException, KeyStoreException,
        UnrecoverableKeyException, KeyManagementException {

    final KeyStore clientStore = KeyStore.getInstance("PKCS12");
    clientStore.load(new FileInputStream(this.certificateResource.getFile()), this.p12PW.toCharArray());

    final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
    keyManagerFactory.init(clientStore, this.p12PW.toCharArray());
    final KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();

    final KeyStore trustStore = KeyStore.getInstance("JKS");
    trustStore.load(new FileInputStream(this.trustStoreResource.getFile()), this.trustStorePW.toCharArray());

    final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    trustManagerFactory.init(trustStore);
    final TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();

    final SSLContext sslContext = SSLContext.getInstance("TLS");
    sslContext.init(keyManagers, trustManagers, new SecureRandom());

    return sslContext;
}

I was following a guideline, which I got from the issuer, which showed how to do this with a cUrl command:

curl --cert Certificate-<id>.pem[:pem_password] https://api.url.com/

So I was basically trying to rebuild this command in java, which is working.

Now for the part that's not working, which is the file upload. Again, I was given a cUrl command which I have to rebuild:

curl --cert Certificate-<id>.pem[:pem_password] -F upload=@<Path_to_file>\DS<PartnerId>_<Timestamp>.XML https://api.url.com/in/upload.php

I tried several things to achieve that:

1. "Normal" Java

Firstly, I tried it with the standard HttpsURLConnection as follows:

final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("192.168.1.25", 3128));
final HttpsURLConnection connection = (HttpsURLConnection) new URL(ARGE_UPLOAD_URL).openConnection(proxy);

connection.setSSLSocketFactory(this.http.createSSLContext().getSocketFactory());
connection.setDoOutput(true);
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");

try (OutputStream outputStream = connection.getOutputStream()) {
    outputStream.write(Files.readAllBytes(new File("src/main/resources/XML/example.xml").toPath()));
}

final InputStream result = connection.getInputStream();

But this always results in java.io.IOException: Server returned HTTP response code: 403 for URL: https://api.url.com/in/upload.php, even though I'm using the same configuration, with which I am able to download from the API.

2. Apache HttpClient

I found some resources claiming that the HttpClient is a lot easier to configure and use, so I gave it a try:

final CloseableHttpClient httpClient = HttpClientBuilder
            .create()
            .setSSLContext(this.http.createSSLContext())
            .setProxy(new HttpHost(InetAddress.getByName(ip), port))
            .build();
final HttpEntity requestEntity = MultipartEntityBuilder
            .create()
            .addBinaryBody("[email protected]", new File("src/main/resources/XML/example.xml")) // Hardcoded for testing
            .build();

final HttpPost post = new HttpPost(https://api.url.com/in/upload.php);
post.setEntity(requestEntity);

try (CloseableHttpResponse response = httpClient.execute(post)) {
    this.logger.info(response.getStatusLine().toString());
    EntityUtils.consume(response.getEntity());
}

Resulting in HTTP/1.1 403 FORBIDDEN

3. HttpClient (FileEntity instead of MultipartEntity)

As a last thing I tried to a FileEntity:

final HttpPost httpPost = new HttpPost(https://api.url.com/in/upload.php);
httpPost.addHeader("content-type", "application/x-www-form-urlencoded;charset=utf-8");

final FileEntity fileEntity = new FileEntity(new File("src/main/resources/XML/example.xml"));
httpPost.setEntity(fileEntity);

System.out.println("executing request " + httpPost.getRequestLine() + httpPost.getConfig());
try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
    final HttpEntity responseEntity = response.getEntity();

    System.out.println("Status: " + response.getStatusLine());
    if (responseEntity != null) {
        System.out.println("Entity: " + EntityUtils.toString(responseEntity));
    }
}

Resulting in Status: HTTP/1.1 403 FORBIDDEN

I just don't understand, how I can be able to download from the API, but not upload to it, despite using exactly the same configuration.

If you need any more information, I'll be happy to provide them.

EDIT

As suggested by oli, I used Fiddler to capture the HTTPS request. This is the result for method 1 (Normal Java):

POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Accept-Charset: utf-8
Accept: */*
Content-Type: application/x-www-form-urlencoded;charset=utf-8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Host: hrbaxml.arbeitsagentur.de
Connection: keep-alive
Content-Length: 6738

And this is the result from the manual upload through Google Chrome:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 6948
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryxZxIJ5h19MEFbZQs
Cookie: cookie_consent=accepted
Host: hrbaxml.arbeitsagentur.de
Origin: https://hrbaxml.arbeitsagentur.de
Referer: https://hrbaxml.arbeitsagentur.de/in/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

EDIT 2

Just to add, this is the result using method 2 (HttpClient with MultipartEntity):

POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Content-Length: 7025
Content-Type: multipart/form-data; boundary=1sZvrqcGe-FuQ3r-_fFgt2SJtZ5_yo7Pfvq_
Host: hrbaxml.arbeitsagentur.de
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_161)
Accept-Encoding: gzip,deflate

--1sZvrqcGe-FuQ3r-_fFgt2SJtZ5_yo7Pfvq_
Content-Disposition: form-data; name="DSV000306700_2018-08-23_09-00-00.xml";    filename="DSV000306700_2018-08-23_09-00-00.xml"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

EDIT 3

I tried copying all the HTTP Headers from the Chrome request, so that my request from Java looks like this:

POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Accept: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Content-Type: multipart/form-data; boundary=1535095530678
Cookie: cookie_consent=accepted
Referer: https://hrbaxml.arbeitsagentur.de/in/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Host: hrbaxml.arbeitsagentur.de
Connection: keep-alive
Content-Length: 6944

--1535095530678
Content-Disposition: form-data; name="uploadFile"; filename="DSV000306700_2018-08-23_09-00-00.xml"
Content-Type: application/xml
Content-Transfer-Encoding: binary

.. xml data ..

--1535095530678--

But still, without success. Any other possible solutions? Maybe it isn't a problem with the upload but something else?

Solution 1

Okay, I finally figured it out. The error was in my POST request, as expected. First, here is the working code (with method 2):

final CloseableHttpClient httpClient = HttpClientBuilder
        .create()
        .setSSLContext(this.http.createSSLContext())
        .setProxy(new HttpHost(InetAddress.getByName(ip), port))
        .build();
final HttpEntity requestEntity = MultipartEntityBuilder
        .create()
        .addBinaryBody("upload", new File("src/main/resources/XML/example.xml")) // Hardcoded for testing
        .build();

final HttpPost post = new HttpPost(https://api.url.com/in/upload.php);
post.setEntity(requestEntity);

try (CloseableHttpResponse response = httpClient.execute(post)) {
    this.logger.info(response.getStatusLine().toString());
    EntityUtils.consume(response.getEntity());
}

As you can see, the only line that changed is .addBinaryBody("upload", new File("src/main/resources/XML/example.xml")). The "upload" is crucial, since the cUrl call I was trying to rebuilt had the -F flag. From the cUrl manual:

(HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multipart/form-data according to RFC 2388.

Each part then consists of a name and the data of the file. The API I'm working with relies on the name being "upload", in order to process the request. Otherwise it doesn't know what to do and returns 403 FORBIDDEN.

Solution 2

I would capture the HTTP request (f.e. with Wireshark) that you send to the server with your Java-Application and compare it with the HTTP request that you send from the browser (yout can easy capture it with the build-in browser tools, try to press F12).

I'm 100% sure that you will see some differences, this is what allways works for me.

EDIT:

There is another possible problem. Please try to add

connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");

or the same one, that your browser sends in your first implementation. Also make sure that you have no problems with your SSL certificate and with encrypt algorithm (you use default one, which one is it in your case). In addition (if nothing helps) you can also check the key length of the negotiationed handshake key.

Comments

[Niby]

Well, that's the problem. Once the manual upload through the browser works (which it does), they do not offer any technical support on the automation.

[Niby]

Thanks for the suggestion. Never tried Wireshark, but it sounds like a really good idea. I'll report back, once I got results.

[Niby]

Wireshark does not seem to be the right tool for this, as this is an HTTPS request.

[Niby]

Thanks, tried it, but still doesn't work. But I got an HTTPS request captured with Fiddler. Please see my edited question.

[John.Chen]

if you could manual upload files, you can use fiddler to capture the network package. Maybe there are some cookies in the http request head.

[Oli]

Have you tried to add the "User-Agent" in your secound example?

[Oli]

I would also set in the first method all the same request properties as in a Chrome request, then it should actually work (if it is not an authentification or authorization or a cetificate problem!).

[Niby]

Yes, changes nothing, unfortunately.

[Niby]

Okay, I'll try that.

[Niby]

I tried it, but still no success. I edited my question.

[Niby]

I don't think I can use Fiddler for a remote host, can I? I used the Google Chrome DevTools to catch the request, but the cookie header only says cookie_consent=accepted. I can't find any more cookies.