Progress of Python requests post

Question

I am uploading a large file using the Python requests package, and I can't find any way to give data back about the progress of the upload. I have seen a number of progress meters for downloading a file, but these will not work for a file upload.

The ideal solution would be some sort of callback method such as:

def progress(percent):
  print percent
r = requests.post(URL, files={'f':hugeFileHandle}, callback=progress)

Thanks in advance for your help :)

Solution 1

requests doesn't support upload streaming e.g.:

import os
import sys
import requests  # pip install requests

class upload_in_chunks(object):
    def __init__(self, filename, chunksize=1 << 13):
        self.filename = filename
        self.chunksize = chunksize
        self.totalsize = os.path.getsize(filename)
        self.readsofar = 0

    def __iter__(self):
        with open(self.filename, 'rb') as file:
            while True:
                data = file.read(self.chunksize)
                if not data:
                    sys.stderr.write("\n")
                    break
                self.readsofar += len(data)
                percent = self.readsofar * 1e2 / self.totalsize
                sys.stderr.write("\r{percent:3.0f}%".format(percent=percent))
                yield data

    def __len__(self):
        return self.totalsize

# XXX fails
r = requests.post("http://httpbin.org/post",
                  data=upload_in_chunks(__file__, chunksize=10))

^{btw, if you don't need to report progress; you could use memory-mapped file to upload large file.}

To workaround it, you could create a file adaptor similar to the one from urllib2 POST progress monitoring:

class IterableToFileAdapter(object):
    def __init__(self, iterable):
        self.iterator = iter(iterable)
        self.length = len(iterable)

    def read(self, size=-1): # TBD: add buffer for `len(data) > size` case
        return next(self.iterator, b'')

    def __len__(self):
        return self.length

Example

it = upload_in_chunks(__file__, 10)
r = requests.post("http://httpbin.org/post", data=IterableToFileAdapter(it))

# pretty print
import json
json.dump(r.json, sys.stdout, indent=4, ensure_ascii=False)

Solution 2

I recommend to use a tool package named requests-toolbelt, which make monitoring upload bytes very easy, like

from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor
import requests

def my_callback(monitor):
    # Your callback function
    print monitor.bytes_read

e = MultipartEncoder(
    fields={'field0': 'value', 'field1': 'value',
            'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
    )
m = MultipartEncoderMonitor(e, my_callback)

r = requests.post('http://httpbin.org/post', data=m,
                  headers={'Content-Type': m.content_type})

And you may want to read this to show a progress bar.

Solution 3

I got it working with the code from here: Simple file upload progressbar in PyQt. I changed it a bit, to use BytesIO instead of StringIO.

class CancelledError(Exception):
    def __init__(self, msg):
        self.msg = msg
        Exception.__init__(self, msg)

    def __str__(self):
        return self.msg

    __repr__ = __str__

class BufferReader(BytesIO):
    def __init__(self, buf=b'',
                 callback=None,
                 cb_args=(),
                 cb_kwargs={}):
        self._callback = callback
        self._cb_args = cb_args
        self._cb_kwargs = cb_kwargs
        self._progress = 0
        self._len = len(buf)
        BytesIO.__init__(self, buf)

    def __len__(self):
        return self._len

    def read(self, n=-1):
        chunk = BytesIO.read(self, n)
        self._progress += int(len(chunk))
        self._cb_kwargs.update({
            'size'    : self._len,
            'progress': self._progress
        })
        if self._callback:
            try:
                self._callback(*self._cb_args, **self._cb_kwargs)
            except: # catches exception from the callback
                raise CancelledError('The upload was cancelled.')
        return chunk


def progress(size=None, progress=None):
    print("{0} / {1}".format(size, progress))


files = {"upfile": ("file.bin", open("file.bin", 'rb').read())}

(data, ctype) = requests.packages.urllib3.filepost.encode_multipart_formdata(files)

headers = {
    "Content-Type": ctype
}

body = BufferReader(data, progress)
requests.post(url, data=body, headers=headers)

The trick is, to generate data and header from the files list manually, using encode_multipart_formdata() from urllib3

Solution 4

I know this is an old question, but I couldn't find an easy answer anywhere else, so hopefully this will help somebody else:

import requests
import tqdm    
with open(file_name, 'rb') as f:
        r = requests.post(url, data=tqdm(f.readlines()))

Solution 5

Usually you would build a streaming datasource (a generator) that reads the file chunked and reports its progress on the way (see kennethreitz/requests#663. This does not work with requests file-api, because requests doesn’t support streaming uploads (see kennethreitz/requests#295) – a file to upload needs to be complete in memory before it starts getting processed.

but requests can stream content from a generator as J.F. Sebastian has proven before, but this generator needs to generate the complete datastream including the multipart encoding and boundaries. This is where poster comes to play.

poster is originally written to be used with pythons urllib2 and supports streaming generation of multipart requests, providing progress indication as it goes along. Posters Homepage provides examples of using it together with urllib2 but you really don’t want to use urllib2. Check out this example-code on how to to HTTP Basic Authentication with urllib2. Horrrrrrrrible.

So we really want to use poster together with requests to do file uploads with tracked progress. And here is how:

# load requests-module, a streamlined http-client lib
import requests

# load posters encode-function
from poster.encode import multipart_encode



# an adapter which makes the multipart-generator issued by poster accessable to requests
# based upon code from http://stackoverflow.com/a/13911048/1659732
class IterableToFileAdapter(object):
    def __init__(self, iterable):
        self.iterator = iter(iterable)
        self.length = iterable.total

    def read(self, size=-1):
        return next(self.iterator, b'')

    def __len__(self):
        return self.length

# define a helper function simulating the interface of posters multipart_encode()-function
# but wrapping its generator with the file-like adapter
def multipart_encode_for_requests(params, boundary=None, cb=None):
    datagen, headers = multipart_encode(params, boundary, cb)
    return IterableToFileAdapter(datagen), headers



# this is your progress callback
def progress(param, current, total):
    if not param:
        return

    # check out http://tcd.netinf.eu/doc/classnilib_1_1encode_1_1MultipartParam.html
    # for a complete list of the properties param provides to you
    print "{0} ({1}) - {2:d}/{3:d} - {4:.2f}%".format(param.name, param.filename, current, total, float(current)/float(total)*100)

# generate headers and gata-generator an a requests-compatible format
# and provide our progress-callback
datagen, headers = multipart_encode_for_requests({
    "input_file": open('recordings/really-large.mp4', "rb"),
    "another_input_file": open('recordings/even-larger.mp4', "rb"),

    "field": "value",
    "another_field": "another_value",
}, cb=progress)

# use the requests-lib to issue a post-request with out data attached
r = requests.post(
    'https://httpbin.org/post',
    auth=('user', 'password'),
    data=datagen,
    headers=headers
)

# show response-code and -body
print r, r.text

View more solutions

Comments

[Blender]

You'd have to implement the progress in hugeFileHandle. I'm not sure why requests doesn't provide a clean way of doing this.

[Robin Begbie]

This worked for the most part, but I found that this just uploaded the contents of the file. Really, what I need is to use requests.post(url, files={'file',fileobj}), and doing this only gives the first chunk of the file using your method

[jfs]

@Robin: the above is a hack that can easily fail. You could try poster instead. It supports progress callbacks and streaming (with known content-length) of multipart/form-data. btw, remove the tick if the answer is not acceptable for your question.

[jfs]

@qarma: If you know a better answer; post it.

[Dima Tisnek]

kennethreitz commented on 10 Jan 2013: Done. github.com/kennethreitz/requests/issues/952

[jfs]

@qarma: I believe you. Could you write a minimal example (with the progress reporting required by OP), test it with a file that is larger than available memory, make sure that the behavior is reasonable: no swapping, the progress report is in real time. I can't delete the accepted answer. I can only provide a link to a better answer.

[Georg]

This would basically be what I need... BUT... is there a way to upload the contents of f.e. file.py in chunks now?

[little]

another question, what if the file is large size ?

[swampfox357]

@Georg, according to the documentation for requests-toolbelt, this should innately support streaming.

[probat]

How is __len__ in class upload_in_chunks(object) automatically called when requests.post() is executed? Is the method __len__ overriding a method in the requests library, I could not find anything in there. If the method is removed, then requests does not upload the file for me.

[jfs]

@probat: In general, the builtin len() function calls the corresponding __len__ method. I don't know whether the answer is applicable to the current requests version.

[probat]

@jfs, sorry I should have been more specific. Using the current version of requests library, one can pass a generator object directly to the post data parameter. I presume at the time this question was originally asked, passing a generator object directly was not possible. Returning to present again, the wrapper class IterableToFileAdapter' is not needed anymore and you can directly pass the class upload_in_chunks` to the data parameter. I see though removing the method def __len__(self): from class upload_in_chunks causes an empty file uploaded using requests.

[probat]

@jfs, continuation of previous comment. If you leave the method that contains the return self.totalsize then the file is correctly uploaded using requests. I cannot figure out why this would be though... The requests documentation says no length is necessary. It seems as if though behind the scenes in the requests source code it is calling len() on the class object upload_in_chunks, but I could not find anything in the source code.

[Naren Babu R]

@jfs i need send the data to files parameter, not data parameter. When I replace data with file, I'm getting this error. TypeError: a bytes-like object is required, not 'upload_in_chunks'