How to upload file with python requests?
Solution 1
If upload_file
is meant to be the file, use:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
and requests
will send a multi-part form POST body with the upload_file
field set to the contents of the file.txt
file.
The filename will be included in the mime header for the specific field:
>>> import requests
>>> open('file.txt', 'wb') # create an empty demo file
<_io.BufferedWriter name='file.txt'>
>>> files = {'upload_file': open('file.txt', 'rb')}
>>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii'))
--c226ce13d09842658ffbd31e0563c6bd
Content-Disposition: form-data; name="upload_file"; filename="file.txt"
--c226ce13d09842658ffbd31e0563c6bd--
Note the filename="file.txt"
parameter.
You can use a tuple for the files
mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:
files = {'upload_file': ('foobar.txt', open('file.txt','rb'), 'text/x-spam')}
This sets an alternative filename and content type, leaving out the optional headers.
If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don't use the files
parameter, just post the file directly as data
. You then may want to set a Content-Type
header too, as none will be set otherwise. See Python requests - POST data from a file.
Solution 2
(2018) the new python requests library has simplified this process, we can use the 'files' variable to signal that we want to upload a multipart-encoded file
url = 'http://httpbin.org/post'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)
r.text
Solution 3
Client Upload
If you want to upload a single file with Python requests
library, then requests lib supports streaming uploads, which allow you to send large files or streams without reading into memory.
with open('massive-body', 'rb') as f:
requests.post('http://some.url/streamed', data=f)
Server Side
Then store the file on the server.py
side such that save the stream into file without loading into the memory. Following is an example with using Flask file uploads.
@app.route("/upload", methods=['POST'])
def upload_file():
from werkzeug.datastructures import FileStorage
FileStorage(request.stream).save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
return 'OK', 200
Or use werkzeug Form Data Parsing as mentioned in a fix for the issue of "large file uploads eating up memory" in order to avoid using memory inefficiently on large files upload (s.t. 22 GiB file in ~60 seconds. Memory usage is constant at about 13 MiB.).
@app.route("/upload", methods=['POST'])
def upload_file():
def custom_stream_factory(total_content_length, filename, content_type, content_length=None):
import tempfile
tmpfile = tempfile.NamedTemporaryFile('wb+', prefix='flaskapp', suffix='.nc')
app.logger.info("start receiving file ... filename => " + str(tmpfile.name))
return tmpfile
import werkzeug, flask
stream, form, files = werkzeug.formparser.parse_form_data(flask.request.environ, stream_factory=custom_stream_factory)
for fil in files.values():
app.logger.info(" ".join(["saved form name", fil.name, "submitted as", fil.filename, "to temporary file", fil.stream.name]))
# Do whatever with stored file at `fil.stream.name`
return 'OK', 200
Solution 4
@martijn-pieters answer is correct, however I wanted to add a bit of context to data=
and also to the other side, in the Flask server, in the case where you are trying to upload files and a JSON.
From the request side, this works as Martijn describes:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
However, on the Flask side (the receiving webserver on the other side of this POST), I had to use form
@app.route("/sftp-upload", methods=["POST"])
def upload_file():
if request.method == "POST":
# the mimetype here isnt application/json
# see here: https://stackoverflow.com/questions/20001229/how-to-get-posted-json-in-flask
body = request.form
print(body) # <- immutable dict
body = request.get_json()
will return nothing. body = request.get_data()
will return a blob containing lots of things like the filename etc.
Here's the bad part: on the client side, changing data={}
to json={}
results in this server not being able to read the KV pairs! As in, this will result in a {} body above:
r = requests.post(url, files=files, json=values). # No!
This is bad because the server does not have control over how the user formats the request; and json=
is going to be the habbit of requests users.
Related videos on Youtube
scichris
Updated on July 15, 2022Comments
-
scichris almost 2 years
I'm performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:
import requests url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post' files={'files': open('file.txt','rb')} values={'upload_file' : 'file.txt' , 'DB':'photcat' , 'OUT':'csv' , 'SHORT':'short'} r=requests.post(url,files=files,data=values)
I'm filling the value of 'upload_file' keyword with my filename, because if I leave it blank, it says
Error - You must select a file to upload!
And now I get
File file.txt of size bytes is uploaded successfully! Query service results: There were 0 lines.
Which comes up only if the file is empty. So I'm stuck as to how to send my file successfully. I know that the file works because if I go to this website and manually fill in the form it returns a nice list of matched objects, which is what I'm after. I'd really appreciate all hints.
Some other threads related (but not answering my problem):
-
William Wino almost 6 yearsHi, How do I send multiple files sharing a same name? Like 'attachment' for example.
-
Martijn Pieters almost 6 years@William: you can use a sequence of 2-value tuples too, which lets you re-use field names:
files = [('attachment', open('attachment1.txt', 'rb')), ('attachment', open('attachment2.txt', 'rb'))]
. Each tuple is a pair of key and value. -
Zaki over 5 yearsAlso you can also use
files={'file':('nameoffile',open('namoffile','rb'),'Content-Type':'text/html','other header'),'file2':('nameoffile2',open('nameoffile2','rb'),'Content-Type':'application/xml','other header')}
but If files={} is used then headers={'Content-Type':'blah blah'} must not be used! -> @martijn-pieters: bacause the multipart/form-data Content-Type must include the boundary value used to deliniate the parts in the post body. Not setting the Content-Type header ensures that requests sets it to the correct value. -
Martijn Pieters over 5 years@zaki: don't conflate the mime type of individual parts with the content type of the POST request. Your syntax in also incorrect, be careful when mixing tuples and dictionaries, and if you are only setting the mime type, just use a string in the 3rd position of a 3-element tuple:
files={'file':('nameoffile', open('nameoffile', 'rb'), text/html'), 'file2': ('nameoffile2',open('nameoffile2', 'rb'), 'application/xml')}
. -
Martijn Pieters over 5 yearsAll this is covered by the requests quickstart and advanced documentation.
-
Matt Messersmith over 5 years@MartijnPieters Doesn't this risk leaking the file? Does
requests
close it? -
Martijn Pieters over 5 years@MattMessersmith: no, it isn't closed. If you want to close the file, use
with open(...) as fobj:
and usefobj
in thefiles
mapping. -
bSr about 5 yearsHow to remove these contents --c226ce13d09842658ffbd31e0563c6bd Content-Disposition: form-data; name="upload_file"; filename="file.txt" --c226ce13d09842658ffbd31e0563c6bd-- from the file while uploading
-
Martijn Pieters about 5 years@bSr: why would you want to remove the mime boundaries and metadata from a multipart/form upload? If you are uploading just the file contents, send it as
data=fileobj
and set a valid content-type header, don't usefiles=
. -
Demetris over 4 yearsDoes requests library automatically close the file?
-
laycat over 4 yearshello, its been awhile since I've used this library. nice question. could you give me and the others a hand by typing lsof | grep "filename" and share your results with us? thanks :)
-
Demetris over 4 yearsWith the use of
lsof
, is seems that the file remains open, or at least, this is how I interpret the following results. Before, running theopen
there is no record inlsof
table about thefilename
. Then after theopen
is executed, multiple records appear withread
access. After executing therequests.post
, the records are still there indicating that the file did not close. -
am.rez over 4 yearswhat is the value of
data
variable? -
Harshit Trivedi over 4 yearsit can be anything like user name,I've just shown how to upload files to REST apis
-
Tommy about 3 years@martijn-pieters just seeing if youd like to include any of this in your answer, and I will delete this; this might be useful for people coming to this answer from a "both sides" of the client and server perspective.
-
guozqzzu over 2 years(2021) If you also need parameters when upload a file, you can add
params
like this:r = requests.post(url,files=files,params={"key":value})
-
jlhasson about 2 yearsYou can find this in the requests documentation here: docs.python-requests.org/en/latest/user/advanced/…
-
Nox about 2 yearsThanks for this answer! I'm looking a bit more into how to upload multiple files using the streaming upload, but most examples are re-using the one you shared with a single
open()
. Would you know how to do that?