Flutter web: How to Upload a Large File?

Question

Is there a way to upload large files to server?

I am using MultipartRequest with MultipartFile like:

  List<int> fileBytes) async {
  var request = new http.MultipartRequest("POST", Uri.parse(url));
  request.files.add(http.MultipartFile.fromBytes(
    'file',
    fileBytes,
    contentType: MediaType('application', 'octet-stream'),
    filename: fileName));
  request.headers.addAll(headers);
  var streamedResponse = await request.send();
  return await http.Response.fromStream(streamedResponse);

and reading the file like:

    html.InputElement uploadInput = html.FileUploadInputElement();
    uploadInput.multiple = false;
    uploadInput.draggable = true;
    uploadInput.click();

    uploadInput.onChange.listen((e) {
      final files = uploadInput.files;
      final file = files[0];

      final reader = new html.FileReader();

      reader.onLoadEnd.listen((e) {
        setState(() {
          _bytesData =
              Base64Decoder().convert(reader.result.toString().split(",").last);
          _selectedFile = _bytesData;
        });
      });

      reader.readAsDataUrl(file);
    });

It is OK for files around 30 MB but for more than that, I am getting Error code: Out of Memory.

Am I doing something wrong? I saw somewhere

MultipartFile.fromBytes will give you some issues on bigger files, as the browser will limit your memory consumption.

And I think his solution is:

There’s a fromStream constructor. Usually, for bigger files, I just use HttpRequest, and put the File object in a FormData instance.

I used MultipartFile and MultipartFile.fromString and both times (for 150 MB file) that happened again. How can I use this solution? or Is there a better way to do that for files more than 500 MB?

Update

Added an answer using Worker. This is not a great solution but I think this might help someone.

Solution 1

Currently, I solved the problem using this approach:

Import:

import 'package:universal_html/html.dart' as html;

Flutter part:

class Upload extends StatefulWidget {
  @override
  _UploadState createState() => _UploadState();
}

class _UploadState extends State<Upload> {
  html.Worker myWorker;
  html.File file;

  _uploadFile() async {
    String _uri = "/upload";

    myWorker.postMessage({"file": file, "uri": _uri});
  }

  _selectFile() {
    html.InputElement uploadInput = html.FileUploadInputElement();
    uploadInput.multiple = false;
    uploadInput.click();

    uploadInput.onChange.listen((e) {
      file = uploadInput.files.first;
    });
  }

  @override
  void initState() {
    myWorker = new html.Worker('upload_worker.js');
    myWorker.onMessage.listen((e) {
      setState(() {
        //progressbar,...
      });
    });

    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        RaisedButton(
          onPressed: _selectFile(),
          child: Text("Select File"),
        ),
        RaisedButton(
          onPressed: _uploadFile(),
          child: Text("Upload"),
        ),
      ],
    );
  }
}

Javascript part:

In the web folder (next to index.html), create the file 'upload_worker.js' .

self.addEventListener('message', async (event) => {
    var file = event.data.file;
    var url = event.data.uri;
    uploadFile(file, url);
});

function uploadFile(file, url) {
    var xhr = new XMLHttpRequest();
    var formdata = new FormData();
    var uploadPercent;

    formdata.append('file', file);

    xhr.upload.addEventListener('progress', function (e) {
        //Use this if you want to have a progress bar
        if (e.lengthComputable) {
            uploadPercent = Math.floor((e.loaded / e.total) * 100);
            postMessage(uploadPercent);
        }
    }, false);
    xhr.onreadystatechange = function () {
        if (xhr.readyState == XMLHttpRequest.DONE) {
            postMessage("done");
        }
    }
    xhr.onerror = function () {
        // only triggers if the request couldn't be made at all
        postMessage("Request failed");
    };

    xhr.open('POST', url, true);

    xhr.send(formdata);
}

Solution 2

I solved the problem using only Dart code: The way to go is to use a chunk uploader. This means to manually send the file in little parts. I send 99MB per request for example. There is already a basic implementation of this online: https://pub.dev/packages/chunked_uploader

You have to get a stream, this is possible with the file_picker or the drop_zone library. I used the drop_zone library because it provides the file picker and the drop zone functionality. In my code the dynamic file objects come from the drop_zone library.

Maybe you have to adjust the chunk uploader functionality depending one your backend. I use a django backend where I wrote a simple view that saves the files. In case of small files it can receive multipart requests with multiple files, in case of large files it can receive chunks and continiues to write a file if a previous chunk was received. Here some parts of my code:

Python backend:

@api_view(["POST"])
def upload(request):
    basePath = config.get("BasePath")
    
    targetFolder = os.path.join(basePath, request.data["taskId"], "input")
    if not os.path.exists(targetFolder):
        os.makedirs(targetFolder)

    for count, file in enumerate(request.FILES.getlist("Your parameter name on server side")):
        path = os.path.join(targetFolder, file.name)
        print(path)
        with open(path, 'ab') as destination:
            for chunk in file.chunks():
                destination.write(chunk)

    return HttpResponse("File(s) uploaded!")

flutter chunk uploader in my version:

import 'dart:async';
import 'dart:html';
import 'dart:math';
import 'package:dio/dio.dart';
import 'package:flutter_dropzone/flutter_dropzone.dart';
import 'package:http/http.dart' as http;

class UploadRequest {
  final Dio dio;
  final String url;
  final String method;
  final String fileKey;
  final Map<String, String>? bodyData;
  final Map<String, String>? headers;
  final CancelToken? cancelToken;
  final dynamic file;
  final Function(double)? onUploadProgress;
  late final int _maxChunkSize;
  int fileSize;
  String fileName;
  late DropzoneViewController controller;

  UploadRequest(
    this.dio, {
    required this.url,
    this.method = "POST",
    this.fileKey = "file",
    this.bodyData = const {},
    this.cancelToken,
    required this.file,
    this.onUploadProgress,
    int maxChunkSize = 1024 * 1024 * 99,
    required this.controller,
    required this.fileSize,
    required this.fileName,
    this.headers
  }) {
    _maxChunkSize = min(fileSize, maxChunkSize);
  }

  Future<Response?> upload() async {
    Response? finalResponse;
    for (int i = 0; i < _chunksCount; i++) {
      final start = _getChunkStart(i);
      print("start is $start");
      final end = _getChunkEnd(i);
      final chunkStream = _getChunkStream(start, end);
      
      
      var request = http.MultipartRequest(
        "POST",
        Uri.parse(url),
      );

      //request.headers.addAll(_getHeaders(start, end));
      request.headers.addAll(headers!);

      //-----add other fields if needed
      request.fields.addAll(bodyData!);

      request.files.add(http.MultipartFile(
        "Your parameter name on server side",
        chunkStream,
        fileSize,
        filename: fileName// + i.toString(),
        )
      );


      //-------Send request
      var resp = await request.send();

      //------Read response
      String result = await resp.stream.bytesToString();

      //-------Your response
      print(result);

      
    }
    return finalResponse;
  }

  Stream<List<int>> _getChunkStream(int start, int end) async* {
    print("reading from $start to $end");
    final reader = FileReader();
    final blob = file.slice(start, end);
    reader.readAsArrayBuffer(blob);
    await reader.onLoad.first;
    yield reader.result as List<int>;
  }


  // Updating total upload progress
  _updateProgress(int chunkIndex, int chunkCurrent, int chunkTotal) {
    int totalUploadedSize = (chunkIndex * _maxChunkSize) + chunkCurrent;
    double totalUploadProgress = totalUploadedSize / fileSize;
    this.onUploadProgress?.call(totalUploadProgress);
  }

  // Returning start byte offset of current chunk
  int _getChunkStart(int chunkIndex) => chunkIndex * _maxChunkSize;

  // Returning end byte offset of current chunk
  int _getChunkEnd(int chunkIndex) =>
      min((chunkIndex + 1) * _maxChunkSize, fileSize);

  // Returning a header map object containing Content-Range
  // https://tools.ietf.org/html/rfc7233#section-2
  Map<String, String> _getHeaders(int start, int end) {
    var header = {'Content-Range': 'bytes $start-${end - 1}/$fileSize'};
    if (headers != null) {
      header.addAll(headers!);
    }
    return header;
  }

  // Returning chunks count based on file size and maximum chunk size
  int get _chunksCount {
    var result = (fileSize / _maxChunkSize).ceil();
    return result;
  }
}

    

Upload code that decides whether to upload multiple files in one request or one file divided to many requests:

//upload the large files

Map<String, String> headers = {
  'Authorization': requester.loginToken!
};

fileUploadView.droppedFiles.sort((a, b) => b.size - a.size);

//calculate the sum of teh files:

double sumInMb = 0;
int divideBy = 1000000;

for (UploadableFile file in fileUploadView.droppedFiles) {
    sumInMb += file.size / divideBy;
}

var dio = Dio();

int uploadedAlready = 0;
for (UploadableFile file in fileUploadView.droppedFiles) {

  if (sumInMb < 99) {
    break;
  }

  var uploadRequest = UploadRequest(
    dio,
    url: requester.backendApi+ "/upload",
    file: file.file,
    controller: fileUploadView.controller!,
    fileSize: file.size,
    fileName: file.name,
    headers: headers,
    bodyData: {
      "taskId": taskId.toString(),
      "user": requester.username!,
    },
  );

  await uploadRequest.upload();

  uploadedAlready++;
  sumInMb -= file.size / divideBy;
}

if (uploadedAlready > 0) {
  fileUploadView.droppedFiles.removeRange(0, uploadedAlready);
}

print("large files uploaded");

// upload the small files

//---Create http package multipart request object
var request = http.MultipartRequest(
  "POST",
  Uri.parse(requester.backendApi+ "/upload"),
);


request.headers.addAll(headers);

//-----add other fields if needed
request.fields["taskId"] = taskId.toString();

print("adding files selected with drop zone");
for (UploadableFile file in fileUploadView.droppedFiles) {

  Stream<List<int>>? stream = fileUploadView.controller?.getFileStream(file.file);

  print("sending " + file.name);

  request.files.add(http.MultipartFile(
      "Your parameter name on server side",
      stream!,
      file.size,
      filename: file.name));
}


//-------Send request
var resp = await request.send();

//------Read response
String result = await resp.stream.bytesToString();

//-------Your response
print(result);

Hopefully this gives you a good overview how I solved the problem.