How to copy InMemoryUploadedFile object to disk

Question

I am trying to catch a file sent with form and perform some operations on it before it will be saved. So I need to create a copy of this file in temp directory, but I don't know how to reach it. Shutil's functions fail to copy this file, since there is no path to it. So is there a way to do this operation in some other way ?

My code :

    image = form.cleaned_data['image']
    temp = os.path.join(settings.PROJECT_PATH, 'tmp')
    sourceFile = image.name # without .name here it wasn't working either
    import shutil
    shutil.copy(sourceFile, temp)

Which raises :

Exception Type: IOError at /
Exception Value: (2, 'No such file or directory')

And the debug :

#  (..)\views.py in function

  67. sourceFile = image.name
  68. import shutil
  69. shutil.copy2(sourceFile, temp) ...

# (..)\Python26\lib\shutil.py in copy2

  92. """Copy data and all stat info ("cp -p src dst").
  93.
  94. The destination may be a directory.
  95.
  96. """
  97. if os.path.isdir(dst):
  98. dst = os.path.join(dst, os.path.basename(src))  
  99. copyfile(src, dst) ... 
 100. copystat(src, dst)
 101.

▼ Local vars
Variable    Value
dst     
u'(..)\\tmp\\myfile.JPG'
src     
u'myfile.JPG'
# (..)\Python26\lib\shutil.py in copyfile

  45. """Copy data from src to dst"""
  46. if _samefile(src, dst):
  47. raise Error, "`%s` and `%s` are the same file" % (src, dst)
  48.
  49. fsrc = None
  50. fdst = None
  51. try:
  52. fsrc = open(src, 'rb') ...
  53. fdst = open(dst, 'wb')
  54. copyfileobj(fsrc, fdst)
  55. finally:
  56. if fdst:
  57. fdst.close()
  58. if fsrc:

▼ Local vars
Variable    Value
dst     
u'(..)\\tmp\\myfile.JPG'
fdst    
None
fsrc    
None
src     
u'myfile.JPG'

Solution 1

This is similar question, it might help.

import os
from django.core.files.storage import default_storage
from django.core.files.base import ContentFile
from django.conf import settings

data = request.FILES['image'] # or self.files['image'] in your form

path = default_storage.save('tmp/somename.mp3', ContentFile(data.read()))
tmp_file = os.path.join(settings.MEDIA_ROOT, path)

Solution 2

As mentioned by @Sławomir Lenart, when uploading large files, you don't want to clog up system memory with a data.read().

From Django docs :

Looping over UploadedFile.chunks() instead of using read() ensures that large files don't overwhelm your system's memory

from django.core.files.storage import default_storage

filename = "whatever.xyz" # received file name
file_obj = request.data['file']

with default_storage.open('tmp/'+filename, 'wb+') as destination:
    for chunk in file_obj.chunks():
        destination.write(chunk)

This will save the file at MEDIA_ROOT/tmp/ as your default_storage will unless told otherwise.

Solution 3

Here is another way to do it with python's mkstemp:

### get the inmemory file
data = request.FILES.get('file') # get the file from the curl

### write the data to a temp file
tup = tempfile.mkstemp() # make a tmp file
f = os.fdopen(tup[0], 'w') # open the tmp file for writing
f.write(data.read()) # write the tmp file
f.close()

### return the path of the file
filepath = tup[1] # get the filepath
return filepath

Solution 4

Your best course of action is to write a custom Upload handler. See the docs . If you add a "file_complete" handler, you can access the file's content regardless of having a memory file or a temp path file. You can also use the "receive_data_chunck" method and write your copy within it.

Regards

Solution 5

This is how I tried to save the file locally

    file_object = request.FILES["document_file"]
    file_name = str(file_object)
    print(f'[INFO] File Name: {file_name}')
    with open(file_name, 'wb+') as f:
        for chunk in file_object.chunks():
            f.write(chunk)

View more solutions

Comments

[Adiyat Mubarak]

Hii, i think hardcoded the "tmp/name.mp3" it will be an issue in cross platform path, I try to improve your code <pre> from django.conf import settings tmp = os.path.join(settings.MEDIA_ROOT, "tmp", data.name) path = default_storage.save(tmp, ContentFile(data.read()))</pre>

[Sławomir Lenart]

issues start when u upload many MB in one file. use buffering then. like data.read(BUF_SIZE) in while loop.

[Davor Lucic]

Many/large is a relative measure and has a number of variables (like, amount of memory, disk size, network type/speed etc), but yes for large files you would probably write to disk in chunks.

[fips]

You can avoid hardcoding the name using: FileField(upload_to='data-files-dir').generate_filename(None‌​, data.name)

[awwester]

i was using this for a while until when a duplicate name is uploaded and django automatically appends a suffix to make the filename unique. That renamed filename was not available at the upload handler level. Other than that this worked very cleanly.

[Daviddd]

Like this I get: FileNotFoundError: [Errno 2] No such file or directory: '<storage>/<path>/<my image name>.png'. Ideas?

[Emile Bergeron]

@Daviddd the file is not where you think it is, and it's irrelevant to this code.

[Kaan Öner]

@Emile Bergeron I am using django_storages as.default_storage. How can I write a file to the media root ?