Apache HttpClient POST Upload File won't work with MultipartEntityBuilder upload

Question

I have an HTML form that has a File Upload in it. I am trying to upload a Word doc through the form using MultipartEntityBuilder, but it is just not working.

The form has several text inputs and 2 File Upload lines in it. If I use the BasicNameValuePair approach and just set those fields as empty Strings, the text fields are submitted correctly. Since I need to upload the files, I changed to MultipartEntityBuilder and can't even get the basic test working again.

The Working Code

HttpClient client = HttpClientBuilder.create().build();
String uploadUrl = "http://somepage.com/upload";

HttpPost httppost = new HttpPost(uploadUrl);

httppost.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httppost.setHeader("Content-Type", "multipart/form-data");
httppost.setHeader("Accept-Encoding", "gzip, deflate");

List<NameValuePair> formparams = new ArrayList<NameValuePair>();

formparams.add(new BasicNameValuePair("testField1", "Value1"));
formparams.add(new BasicNameValuePair("testField2", "Value2"));
formparams.add(new BasicNameValuePair("fileField1", ""));
formparams.add(new BasicNameValuePair("fileField2", ""));

httppost.setEntity(new UrlEncodedFormEntity(formparams, Consts.UTF_8));

HttpResponse response = client.execute(httppost, httpContext);

Trying MultipartEntityBuilder

// Everything up to and including the Headers remain the same

MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);

StringBody sb = new StringBody("Value1", ContentType.TEXT_PLAIN);
builder.addPart("textField1", sb);

sb = new StringBody("Value2", ContentType.TEXT_PLAIN);
builder.addPart("textField2", sb);

FileBody fb = new FileBody(new File("path to file"), ContentType.DEFAULT_BINARY);
builder.addPart("fileField1", fb);

fb = new FileBody(new Field("path to file2"), ContentType.DEFAULT_BINARY);
builder.addPart("fileField2", fb);

httppost.setEntity(builder.build());

HttpResponse response = client.execute(httppost, httpContext);

The form is hosted by Oracle and get back the error:

An unexpected error occurred: ORA-06501: PL/SQL: program error

I have also tried these other approaches:

builder.addTextBody("textField1", "Value1");
builder.addTextBody("textField2", "Value2");

File file1 = new File("path to file1");
builder.addBinaryBody("fileField1", file1, ContentType.DEFAULT_BINARY, file1.getName());

File file2 = new File("path to file2");
builder.addBinaryBody("fileField2", file2, ContentType.DEFAULT_BINARY, file2.getName());

As well as trying to mimic my first test by putting everything in the builder.addTextBody() like my original working test, but still get the same result.

Logging

As per @oleg request, I have posted the wire log from this call:

DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: default
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: SSO_TIMEOUT_ID][value: v1.0~83BA4EF3DA76C07B55F93B5C5D65F90947314693035F046BFCC21BCD37F8B95284E732E711971532B182F90AE461E320FCCC74452BAF4A16FB6E5EFA5F86985B26C95D30001D9ACC3BE8E9D2786B1CD38A79788FC7623FCE06C84266C234638182D44786E4971B53EBFC25FD3B7A565F][domain: isomething.com][path: /][expiry: null] match [wwwdev.isomething.com:80/portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept]
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: cust_portal][value: 9.0.3+en-us+us+AMERICA+12955AE0CEA20F3CE050558C15F00BD2+84676D40A18761D45DEBA039A78FF868CA9B49F2DEA2D283DE61561CE0F547D3A27C643219F1E6C867CF150CDEA69AE9844407F570B4BBD967491098CECEEA836861C9FF1F06AF7929814DF3C55426F1C2E73C91B219801B][domain: wwwdev.isomething.com][path: /][expiry: null] match [wwwdev.isomething.com:80/portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept]
DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
DEBUG [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection request: [route: {}->http://wwwdev.isomething.com:80][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
DEBUG [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection leased: [id: 4][route: {}->http://wwwdev.isomething.com:80][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
DEBUG [org.apache.http.impl.execchain.MainClientExec] Opening connection {}->http://wwwdev.isomething.com:80
DEBUG [org.apache.http.impl.conn.DefaultHttpClientConnectionOperator] Connecting to wwwdev.isomething.com/141.146.161.39:80
DEBUG [org.apache.http.impl.conn.DefaultHttpClientConnectionOperator] Connection established 10.0.0.2:49827<->141.146.161.39:80
DEBUG [org.apache.http.impl.execchain.MainClientExec] Executing request POST /portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept HTTP/1.1
DEBUG [org.apache.http.impl.execchain.MainClientExec] Target auth state: UNCHALLENGED
DEBUG [org.apache.http.impl.execchain.MainClientExec] Proxy auth state: UNCHALLENGED
DEBUG [org.apache.http.headers] http-outgoing-4 >> POST /portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept HTTP/1.1
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
DEBUG [org.apache.http.headers] http-outgoing-4 >> Origin: http://wwwdev.isomething.com
DEBUG [org.apache.http.headers] http-outgoing-4 >> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36
DEBUG [org.apache.http.headers] http-outgoing-4 >> Referer: http://wwwdev.isomething.com/portal/page/cust_portal/SOM_PGR/ReportManager2/ReportManager/SOM_Reports_MTC/SOM_FILE_CLERK_ADD_REPORT?np_report_number=MTC01234567&np_project_number=3141307&np_country=US&np_customer_number=110960&np_module=URM&np_customer_name=MY TEST CENTRAL TEST ACCOUNT
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept-Encoding: gzip, deflate
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept-Language: en-US,en;q=0.8
DEBUG [org.apache.http.headers] http-outgoing-4 >> DNT: 1
DEBUG [org.apache.http.headers] http-outgoing-4 >> Content-Length: 49234
DEBUG [org.apache.http.headers] http-outgoing-4 >> Content-Type: multipart/form-data; boundary=_I6HZ69Fquh0RHe-DNWGs9tfX48pemmkOYs
DEBUG [org.apache.http.headers] http-outgoing-4 >> Host: wwwdev.isomething.com
DEBUG [org.apache.http.headers] http-outgoing-4 >> Connection: Keep-Alive
DEBUG [org.apache.http.headers] http-outgoing-4 >> Cookie: SSO_TIMEOUT_ID=v1.0~83BA4EF3DA76C07B55F93B5C5D65F90947314693035F046BFCC21BCD37F8B95284E732E711971532B182F90AE461E320FCCC74452BAF4A16FB6E5EFA5F86985B26C95D30001D9ACC3BE8E9D2786B1CD38A79788FC7623FCE06C84266C234638182D44786E4971B53EBFC25FD3B7A565F; cust_portal=9.0.3+en-us+us+AMERICA+12955AE0CEA20F3CE050558C15F00BD2+84676D40A18761D45DEBA039A78FF868CA9B49F2DEA2D283DE61561CE0F547D3A27C643219F1E6C867CF150CDEA69AE9844407F570B4BBD967491098CECEEA836861C9FF1F06AF7929814DF3C55426F1C2E73C91B219801B
DEBUG [org.apache.http.headers] http-outgoing-4 << HTTP/1.1 200 OK
DEBUG [org.apache.http.headers] http-outgoing-4 << Cache-Control: max-age=0
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Type: text/html; charset=UTF-8
DEBUG [org.apache.http.headers] http-outgoing-4 << Set-Cookie: SSO_TIMEOUT_ID=v1.0~83BA4EF3DA76C07B68FCC2530F599A0EC0C64F824C7AE3F72C95A8D07625F4915248DB9B7D40E28DF645BB373ADAE5E39B2A539F98F48507192E9993DAFEDE4D30331E7912A944E0A9C203BD851C0C0D7DCBC672186F9DF652220BC26B85C327A81DE6656E5D73550FCD2EA1BA53552F; domain=.isomething.com; path=/
DEBUG [org.apache.http.headers] http-outgoing-4 << Connection: Keep-Alive
DEBUG [org.apache.http.headers] http-outgoing-4 << Keep-Alive: timeout=5, max=999
DEBUG [org.apache.http.headers] http-outgoing-4 << Server: Oracle-Application-Server-10g/10.1.2.3.0 Oracle-HTTP-Server OracleAS-Web-Cache-10g/10.1.2.3.2 (N;ecid=100569954162,0)
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Length: 5134
DEBUG [org.apache.http.headers] http-outgoing-4 << Date: Tue, 31 Mar 2015 13:06:53 GMT
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Location: /servlet/RepositoryServlet/cust_portal/!CUST_PORTAL.wwa_app_module.accept
DEBUG [org.apache.http.impl.execchain.MainClientExec] Connection can be kept alive for 5000 MILLISECONDS
DEBUG [org.apache.http.client.protocol.ResponseProcessCookies] Cookie accepted [SSO_TIMEOUT_ID="v1.0~83BA4EF3DA76C07B68FCC2530F599A0EC0C64F824C7AE3F72C95A8D07625F4915248DB9B7D40E28DF645BB373ADAE5E...", version:0, domain:isomething.com, path:/, expiry:null]

Solution 1

Some time ago I spent lots of time trying to submit multipart form with file. Finally problem were solved by adding boundary. As I understood after some debugging server wasn't able to recognize parts without it.

String boundary = "---------------"+UUID.randomUUID().toString();
multipartEntityBuilder.setBoundary(boundary);
...
HttpPost request = new HttpPost(uriBuilder.build());
request.setHeader("Content-Type", ContentType.MULTIPART_FORM_DATA.getMimeType()+";boundary="+boundary);
...
multipartEntityBuilder.addBinaryBody("document[file]", dataBytes, ContentType.APPLICATION_OCTET_STREAM, name);

Solution 2

Try removing this line

httppost.setHeader("Content-Type", "multipart/form-data");

Updated

as expected the application is sending invalid Content-Type header

[org.apache.http.headers] http-outgoing-5 >> Content-Type: multipart/form-data

Please do not set Content-Type manually and let HttpClient generate it for you based on properties of the HttpEntity enclosed in the request

Comments

[ok2c]

HttpClient generates correct Content-Type header based on properties of the enclosed request entity by default. One generally should never need to set Content-Type header on request message.

[Stan]

The reason for that was adding boundary parameter to header value since it wasn't added by the HttpClient.

[ok2c]

That's wrong. The boundary element is populated automatically by HttpClient

[Stan]

I checked with network monitor tool (don't remember exact name), it wasn't for sure. Maybe was problem of some concrete version of HttpClient.

[ok2c]

I doubt that. Anyhow, absence of the boundary attribute in the Content-Type header indeed appears to be the most likely cause of the problem

[Ascalonian]

This did not do anything different

[Ascalonian]

Unfortunately, that didn't do anything either

[Ascalonian]

Added to the question :-)

[Ascalonian]

I did not set the Content Type. That is what the HttpClient is generating. I removed all lines setting the content type in the header. Only place I do anything with Content Type is in the StringBody and FileBody. I updated the logs with the latest and greatest of what is returned.

[ok2c]

Then, it must be green men from Mars messing with your computer. Because this is what HttpClient is generating by default: hc.apache.org/httpcomponents-client-4.4.x/httpmime/xref/org/‌​…

[Ascalonian]

According to the code, if ContentType is null, it goes to "multipart/form-data" line 209. What do you think the content type should be?

[ok2c]

That entirely depends on 'params', which should always have a boundary attribute: ContentType.create("multipart/" + DEFAULT_SUBTYPE, params)

[Ola Aronsson]

..and of course all fields in the MultipartMap should be UTF-8-encoded and I added multipartBuilderInner.setCharset(StandardCharsets.UTF_8); pre-build..