Send file using POST from a Python script
Solution 1
From: https://requests.readthedocs.io/en/latest/user/quickstart/#post-a-multipart-encoded-file
Requests makes it very simple to upload Multipart-encoded files:
with open('report.xls', 'rb') as f:
r = requests.post('http://httpbin.org/post', files={'report.xls': f})
That's it. I'm not joking - this is one line of code. The file was sent. Let's check:
>>> r.text
{
"origin": "179.13.100.4",
"files": {
"report.xls": "<censored...binary...data>"
},
"form": {},
"url": "http://httpbin.org/post",
"args": {},
"headers": {
"Content-Length": "3196",
"Accept-Encoding": "identity, deflate, compress, gzip",
"Accept": "*/*",
"User-Agent": "python-requests/0.8.0",
"Host": "httpbin.org:80",
"Content-Type": "multipart/form-data; boundary=127.0.0.1.502.21746.1321131593.786.1"
},
"data": ""
}
Solution 2
Yes. You'd use the urllib2
module, and encode using the multipart/form-data
content type. Here is some sample code to get you started -- it's a bit more than just file uploading, but you should be able to read through it and see how it works:
user_agent = "image uploader"
default_message = "Image $current of $total"
import logging
import os
from os.path import abspath, isabs, isdir, isfile, join
import random
import string
import sys
import mimetypes
import urllib2
import httplib
import time
import re
def random_string (length):
return ''.join (random.choice (string.letters) for ii in range (length + 1))
def encode_multipart_data (data, files):
boundary = random_string (30)
def get_content_type (filename):
return mimetypes.guess_type (filename)[0] or 'application/octet-stream'
def encode_field (field_name):
return ('--' + boundary,
'Content-Disposition: form-data; name="%s"' % field_name,
'', str (data [field_name]))
def encode_file (field_name):
filename = files [field_name]
return ('--' + boundary,
'Content-Disposition: form-data; name="%s"; filename="%s"' % (field_name, filename),
'Content-Type: %s' % get_content_type(filename),
'', open (filename, 'rb').read ())
lines = []
for name in data:
lines.extend (encode_field (name))
for name in files:
lines.extend (encode_file (name))
lines.extend (('--%s--' % boundary, ''))
body = '\r\n'.join (lines)
headers = {'content-type': 'multipart/form-data; boundary=' + boundary,
'content-length': str (len (body))}
return body, headers
def send_post (url, data, files):
req = urllib2.Request (url)
connection = httplib.HTTPConnection (req.get_host ())
connection.request ('POST', req.get_selector (),
*encode_multipart_data (data, files))
response = connection.getresponse ()
logging.debug ('response = %s', response.read ())
logging.debug ('Code: %s %s', response.status, response.reason)
def make_upload_file (server, thread, delay = 15, message = None,
username = None, email = None, password = None):
delay = max (int (delay or '0'), 15)
def upload_file (path, current, total):
assert isabs (path)
assert isfile (path)
logging.debug ('Uploading %r to %r', path, server)
message_template = string.Template (message or default_message)
data = {'MAX_FILE_SIZE': '3145728',
'sub': '',
'mode': 'regist',
'com': message_template.safe_substitute (current = current, total = total),
'resto': thread,
'name': username or '',
'email': email or '',
'pwd': password or random_string (20),}
files = {'upfile': path}
send_post (server, data, files)
logging.info ('Uploaded %r', path)
rand_delay = random.randint (delay, delay + 5)
logging.debug ('Sleeping for %.2f seconds------------------------------\n\n', rand_delay)
time.sleep (rand_delay)
return upload_file
def upload_directory (path, upload_file):
assert isabs (path)
assert isdir (path)
matching_filenames = []
file_matcher = re.compile (r'\.(?:jpe?g|gif|png)$', re.IGNORECASE)
for dirpath, dirnames, filenames in os.walk (path):
for name in filenames:
file_path = join (dirpath, name)
logging.debug ('Testing file_path %r', file_path)
if file_matcher.search (file_path):
matching_filenames.append (file_path)
else:
logging.info ('Ignoring non-image file %r', path)
total_count = len (matching_filenames)
for index, file_path in enumerate (matching_filenames):
upload_file (file_path, index + 1, total_count)
def run_upload (options, paths):
upload_file = make_upload_file (**options)
for arg in paths:
path = abspath (arg)
if isdir (path):
upload_directory (path, upload_file)
elif isfile (path):
upload_file (path)
else:
logging.error ('No such path: %r' % path)
logging.info ('Done!')
Solution 3
Looks like python requests does not handle extremely large multi-part files.
The documentation recommends you look into requests-toolbelt
.
Here's the pertinent page from their documentation.
Solution 4
The only thing that stops you from using urlopen directly on a file object is the fact that the builtin file object lacks a len definition. A simple way is to create a subclass, which provides urlopen with the correct file. I have also modified the Content-Type header in the file below.
import os
import urllib2
class EnhancedFile(file):
def __init__(self, *args, **keyws):
file.__init__(self, *args, **keyws)
def __len__(self):
return int(os.fstat(self.fileno())[6])
theFile = EnhancedFile('a.xml', 'r')
theUrl = "http://example.com/abcde"
theHeaders= {'Content-Type': 'text/xml'}
theRequest = urllib2.Request(theUrl, theFile, theHeaders)
response = urllib2.urlopen(theRequest)
theFile.close()
for line in response:
print line
Solution 5
Chris Atlee's poster library works really well for this (particularly the convenience function poster.encode.multipart_encode()
). As a bonus, it supports streaming of large files without loading an entire file into memory. See also Python issue 3244.
Related videos on Youtube
Comments
-
readonly about 3 years
Is there a way to send a file using POST from a Python script?
-
amit kumar over 13 yearsOn python 2.6.6 I was getting an error in Multipart boundary parsing while using this code on Windows. I had to change from string.letters to string.ascii_letters as discussed at stackoverflow.com/questions/2823316/… for this to work. The requirement on boundary is discussed here: stackoverflow.com/questions/147451/…
-
dland over 12 yearsThere are no examples that show how to deal with file uploads.
-
tabdulradi over 12 yearscalling run_upload ({'server':'', 'thread':''}, paths=['/path/to/file.txt']) causes error in this line: upload_file (path) because "upload file" requires 3 parameters so I replaces it with this line upload_file (path, 1, 1)
-
RayLuo about 11 years@robert I test your code in Python2.7 but it doesn't work. urlopen(Request(theUrl, theFile, ...)) merely encodes the content of file as if a normal post but can not specify the correct form field. I even try variant urlopen(theUrl, urlencode({'serverside_field_name': EnhancedFile('my_file.txt')})), it uploads a file but (of course!) with incorrect content as <open file 'my_file.txt', mode 'r' at 0x00D6B718>. Did I miss something?
-
Akshay Patil almost 11 yearsThanks for the answer . By using the above code I had transferred 2.2 GB raw image file by using PUT request into the webserver.
-
Niks Jain over 10 yearsI'm trying the same thing & its working fine if file size is less than ~1.5 MB. else its throwing an error.. please have look at here.
-
TaraGurung almost 9 yearswhat am trying to do is login to some site using request which i have done successfully but now i want to upload a video after logging in and the form has a different fields to be filled before submission. So how should I pass those values like videos description,videos title etc
-
Hjulle over 8 yearsYou'd probably want to do
with open('report.xls', 'rb') as f: r = requests.post('http://httpbin.org/post', files={'report.xls': f})
instead, so it closes the file again after opening. -
jlr over 7 yearsLink is outdated + no inlined example.
-
bmoran over 7 yearsThis answer should be updated to include Hjulle's suggestion of using the context manager to ensure file is closed.
-
pdc about 7 yearsIt has since moved to github.com/httplib2/httplib2. On the other hand, nowadays I would probably recommend
requests
instead. -
Saurabh Jain almost 7 yearsthis is not working for me, it says '405 method not allowed.' with open(file_path, 'rb') as f: response = requests.post(url=url, data=f, auth=HTTPBasicAuth(username=id, password=password) )
-
Santhosh about 6 yearsI have a raw
.jpg
format file stored asndarray
variable, how can I upload this to server? -
Kavi Vaidya over 5 yearsCan we send pdf's using this? Because isn't this only for text files?
-
Chiefir over 5 yearsthis code provides to a memory leak - you forgot to
close()
a file. -
obotezat about 3 yearsIt is generally a good practice to write the comments in english.