Tracking progress of multipart file upload using OKHTTP

Question

I am trying to implement a a progress bar to indicate the progress of a multipart file upload.

I have read from a comment on this answer - https://stackoverflow.com/a/24285633/1022454 that I have to wrap the sink passed to the RequestBody and provide a callback that tracks the bytes moved.

I have created a custom RequestBody and wrapped the sink with a CustomSink class, however through debugging I can see that the bytes are being written by RealBufferedSink ln 44 and the custom sink write method is only run once, not allowing me to track the bytes moved.

    private class CustomRequestBody extends RequestBody {

    MediaType contentType;
    byte[] content;

    private CustomRequestBody(final MediaType contentType, final byte[] content) {
        this.contentType = contentType;
        this.content = content;
    }

    @Override
    public MediaType contentType() {
        return contentType;
    }

    @Override
    public long contentLength() {
        return content.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException {
        CustomSink customSink = new CustomSink(sink);
        customSink.write(content);

    }
}


private class CustomSink implements BufferedSink {

    private static final String TAG = "CUSTOM_SINK";

    BufferedSink bufferedSink;

    private CustomSink(BufferedSink bufferedSink) {
        this.bufferedSink = bufferedSink;
    }

    @Override
    public void write(Buffer source, long byteCount) throws IOException {
        Log.d(TAG, "source size: " + source.size() + " bytecount" + byteCount);
        bufferedSink.write(source, byteCount);
    }

    @Override
    public void flush() throws IOException {
        bufferedSink.flush();
    }

    @Override
    public Timeout timeout() {
        return bufferedSink.timeout();
    }

    @Override
    public void close() throws IOException {
        bufferedSink.close();
    }

    @Override
    public Buffer buffer() {
        return bufferedSink.buffer();
    }

    @Override
    public BufferedSink write(ByteString byteString) throws IOException {
        return bufferedSink.write(byteString);
    }

    @Override
    public BufferedSink write(byte[] source) throws IOException {
        return bufferedSink.write(source);
    }

    @Override
    public BufferedSink write(byte[] source, int offset, int byteCount) throws IOException {
        return bufferedSink.write(source, offset, byteCount);
    }

    @Override
    public long writeAll(Source source) throws IOException {
        return bufferedSink.writeAll(source);
    }

    @Override
    public BufferedSink writeUtf8(String string) throws IOException {
        return bufferedSink.writeUtf8(string);
    }

    @Override
    public BufferedSink writeString(String string, Charset charset) throws IOException {
        return bufferedSink.writeString(string, charset);
    }

    @Override
    public BufferedSink writeByte(int b) throws IOException {
        return bufferedSink.writeByte(b);
    }

    @Override
    public BufferedSink writeShort(int s) throws IOException {
        return bufferedSink.writeShort(s);
    }

    @Override
    public BufferedSink writeShortLe(int s) throws IOException {
        return bufferedSink.writeShortLe(s);
    }

    @Override
    public BufferedSink writeInt(int i) throws IOException {
        return bufferedSink.writeInt(i);
    }

    @Override
    public BufferedSink writeIntLe(int i) throws IOException {
        return bufferedSink.writeIntLe(i);
    }

    @Override
    public BufferedSink writeLong(long v) throws IOException {
        return bufferedSink.writeLong(v);
    }

    @Override
    public BufferedSink writeLongLe(long v) throws IOException {
        return bufferedSink.writeLongLe(v);
    }

    @Override
    public BufferedSink emitCompleteSegments() throws IOException {
        return bufferedSink.emitCompleteSegments();
    }

    @Override
    public OutputStream outputStream() {
        return bufferedSink.outputStream();
    }
}

Does anybody have an example of how I would go about doing this?

Solution 1

You have to create a custom RequestBody and override writeTo method, and there you have to send your files down the sink in segments. It is very important that you flush the sink after each segment, otherwise your progress bar will fill up quickly without the file being actually sent over the network, because the contents will stay in the sink (which acts like a buffer).

public class CountingFileRequestBody extends RequestBody {

    private static final int SEGMENT_SIZE = 2048; // okio.Segment.SIZE

    private final File file;
    private final ProgressListener listener;
    private final String contentType;

    public CountingFileRequestBody(File file, String contentType, ProgressListener listener) {
        this.file = file;
        this.contentType = contentType;
        this.listener = listener;
    }

    @Override
    public long contentLength() {
        return file.length();
    }

    @Override
    public MediaType contentType() {
        return MediaType.parse(contentType);
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException {
        Source source = null;
        try {
            source = Okio.source(file);
            long total = 0;
            long read;

            while ((read = source.read(sink.buffer(), SEGMENT_SIZE)) != -1) {
                total += read;
                sink.flush();
                this.listener.transferred(total);

            }
        } finally {
            Util.closeQuietly(source);
        }
    }

    public interface ProgressListener {
        void transferred(long num);
    }

}

You can find a complete implementation that supports displaying progress in an AdapterView and also cancelling uploads at my gist: https://gist.github.com/eduardb/dd2dc530afd37108e1ac

Solution 2

• We just need to create a custom RequestBody, no need to implement custom BufferedSink. We can allocate Okio buffer to read from image file, and connect this buffer to sink.

For an example, please see the below createCustomRequestBody function

public static RequestBody createCustomRequestBody(final MediaType contentType, final File file) {
    return new RequestBody() {
        @Override public MediaType contentType() {
            return contentType;
        }
        @Override public long contentLength() {
            return file.length();
        }
        @Override public void writeTo(BufferedSink sink) throws IOException {
            Source source = null;
            try {
                source = Okio.source(file);
                //sink.writeAll(source);
                Buffer buf = new Buffer();
                Long remaining = contentLength();
                for (long readCount; (readCount = source.read(buf, 2048)) != -1; ) {
                    sink.write(buf, readCount);
                    Log.d(TAG, "source size: " + contentLength() + " remaining bytes: " + (remaining -= readCount));
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    };
}

• to use -

  .addPart(
      Headers.of("Content-Disposition", "form-data; name=\"image\""),
      createCustomRequestBody(MediaType.parse("image/png"), new File("test.jpg")))
  .build()
  

Solution 3

This thing works great!

Gradle

dependencies {
  compile 'io.github.lizhangqu:coreprogress:1.0.2'
}

//wrap your original request body with progress
RequestBody requestBody = ProgressHelper.withProgress(body, new ProgressUIListener()....} 

Full example code here https://github.com/lizhangqu/CoreProgress

Solution 4

This might be helpful for Kotlin users,I wrote an extension function on java.io.File.

import okhttp3.MediaType
import okhttp3.RequestBody
import okio.BufferedSink
import okio.source
import java.io.File

fun File.asProgressRequestBody(
    contentType: MediaType? = null,
    onProgress: (percent: Float) -> Unit
): RequestBody {
    return object : RequestBody() {
        override fun contentType() = contentType

        override fun contentLength() = length()

        override fun writeTo(sink: BufferedSink) {
            source().use { source ->
                var total: Long = 0
                var read: Long = 0
                while (source.read(sink.buffer, 2048).apply {
                        read = this
                    } != -1L) {
                    total += read
                    sink.flush()
                    onProgress.invoke(
                        (total.toFloat() / length())*100
                    )
                }
            }


        }
    }
}

you can use it in Retrofit as well as OkHttp to create a Multipart request body as follows,

fun file2MultiPartBody(myFile: File,key:String,onProgress:(percent:Float)->Unit): MultipartBody.Part {
    val requestBody = myFile.asProgressRequestBody("application/octet-stream".toMediaType(),onProgress)
    return MultipartBody.Part.createFormData(key, myFile.name, requestBody)
}

View more solutions

Comments

[coalmee]

This doesn't seem to work if uploading small files over a slow network connection. See github.com/square/okhttp/issues/1078. Is there a solution for this case?

[Sree]

@Edy Bolos Is there a way to use it in conjunction with RxJava & Observable?

[Eduard B.]

To answer your question: everything can be wrapped in an Observable :) But I'll have to leave someone else to do it. My only suggestion is to maybe use a BehaviorSubject to emit the progress value in UploadsHandler

[Simon]

As what a previous commenter has said. Small files don't get a progress even on a bad connection. This is regarding the network interface send buffer being so high as to swallow the entire file and okHttp will report it 100% uploaded. Using this fixes this issue. gist.github.com/slightfoot/00a26683ea68856ceb50e26c7d8a47d0

[kinsley kajiva]

can you guys define what you mean by small as in is it 10MB or 2MB or what...??

[Sagar Nayak]

i want to create a custom request body that will be generic. so it can send a pojo or a file. so i will be using toRequestBody extension function. can you please help to work with that.

[lasec0203]

This won't work as expected due to the reason explained in the accepted answer. It is very important that you flush the sink after each segment, otherwise your progress bar will fill up quickly without the file being actually sent over the network

[Adrian Mole]

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review

[DevinM]

This works BUT logs show when uploading that it runs through the first time before uploading and then again after causing the progress bar to complete and then complete a second time (when it actually uploads the file).