HttpUrlConnection multipart file upload with progressBar

Question

I want to check progress of uploading file by HttpUrlConnection. How I can do this? I've tried to calculate bytes when writing data in OutputStream but it's wrong, cause real uploading happens only when I call conn.getInputStream(), so I need somehow to check inputStream. Here is my code:

public static void uploadMovie(final HashMap<String, String> dataSource, final OnLoadFinishedListener finishedListener, final ProgressListener progressListener) {
  if (finishedListener != null) {
    new Thread(new Runnable() {
       public void run() {
         try {

              String boundary = getMD5(dataSource.size()+String.valueOf(System.currentTimeMillis()));
              MultipartEntityBuilder multipartEntity = MultipartEntityBuilder.create();
              multipartEntity.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);    
              multipartEntity.setCharset(Charset.forName("UTF-8"));

              for (String key : dataSource.keySet()) {
                 if (key.equals(MoviesFragmentAdd.USERFILE)) {
                    FileBody  userFile = new FileBody(new File(dataSource.get(key)));
                    multipartEntity.addPart(key, userFile);
                    continue;
                 }
                 multipartEntity.addPart(key, new StringBody(dataSource.get(key),ContentType.APPLICATION_JSON));
              }

              HttpEntity entity = multipartEntity.build();
              HttpURLConnection conn = (HttpsURLConnection) new URL(URL_API + "/video/addForm/").openConnection();
              conn.setUseCaches(false);
              conn.setDoOutput(true);
              conn.setDoInput(true);
              conn.setRequestMethod("POST");
              conn.setRequestProperty("Accept-Charset", "UTF-8");
              conn.setRequestProperty("Connection", "Keep-Alive");
              conn.setRequestProperty("Cache-Control", "no-cache");
              conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
              conn.setRequestProperty("Content-length", entity.getContentLength() + "");
              conn.setRequestProperty(entity.getContentType().getName(),entity.getContentType().getValue());

              OutputStream os = conn.getOutputStream();
              entity.writeTo(os);
              os.close();

              //Real upload starting here -->>

              BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));

              //<<--

              JsonObject request = (JsonObject) gparser.parse(in.readLine());
              if (!request.get("error").getAsBoolean()) {
              //do something
              }
              conn.disconnect(); 

           } catch (Exception e) {
            e.printStackTrace();
           }
         }
    }).start();

  }
}

Solution

Because you have to deal with upload, I'd suppose most time is taken when doing entity.writeTo(os);. Maybe the first contact to the server takes some time as well (DNS resolution, SSL-handshake, ...). The markers you set for "the real upload" are not correct IMO.

Now it depends on your Multipart-library, whether you can intercept writeTo. If it is clever and resource-efficient, it's iterating over the parts and streams the content one-by-one to the output stream. If not, and the .build() operation is creating a big fat byte[], then you could take this array, stream it in chunks to the server and tell your user how many percent of the upload is already done.

From a resource perspective I'd prefer not really knowing what happens. But if feedback is that important and if the movies are only a few megabytes in size, you could stream the Multipart-Entity first to a ByteArrayOutputStream and then write little chunks of the created byte-array to the server while notifying your user about progress. The following code is not validated or tested (you can see it as pseudo-code):

ByteArrayOutputStream baos = new ByteArrayOutputStream();
entity.writeTo(baos);
baos.close();
byte[] payload = baos.toByteArray();
baos = null;

OutputStream os = conn.getOutputStream();

int totalSize = payload.length;
int bytesTransferred = 0;
int chunkSize = 2000;

while (bytesTransferred < totalSize) {
    int nextChunkSize = totalSize - bytesTransferred;
    if (nextChunkSize > chunkSize) {
        nextChunkSize = chunkSize;
    }
    os.write(payload, bytesTransferred, nextChunkSize); // TODO check outcome!
    bytesTransferred += nextChunkSize;

    // Here you can call the method which updates progress
    // be sure to wrap it so UI-updates are done on the main thread!
    updateProgressInfo(100 * bytesTransferred / totalSize);
}
os.close();

A more elegant way would be to write an intercepting OutputStream which registers progress and delegates the real write-operations to the underlaying "real" OutputStream.

Edit

@whizzzkey wrote:

I've re-checked it many times - entity.writeTo(os) DOESN'T do a real upload, it does conn.getResponseCode() or conn.getInputStream()

Now it's clear. HttpURLConnection is buffering your upload data, because it doesn't know the content-length. You've set the header 'Content-length', but oviously this is ignored by HUC. You have to call

conn.setFixedLengthStreamingMode(entity.getContentLength());

Then you should better remove the call to conn.setRequestProperty("Content-length", entity.getContentLength() + "");

In this case, HUC can write the headers and entity.writeTo(os) can really stream the data to the server. Otherwise the buffered data is sent when HUC knows how many bytes will be transferred. So in fact, getInputStream() tells HUC that you're finished, but before really reading the response, all the collected data has to be sent to the server.

I wouldn't recommend changing your code, but for those of you who don't know the exact size of the transferred data (in bytes, not characters!!), you can tell HUC that it should transfer the data in chunks without setting the exact content-length:

conn.setChunkedStreamingMode(-1); // use default chunk size