Splitting a file before upload?

Question

On a webpage, is it possible to split large files into chunks before the file is uploaded to the server? For example, split a 10MB file into 1MB chunks, and upload one chunk at a time while showing a progress bar?

It sounds like JavaScript doesn't have any file manipulation abilities, but what about Flash and Java applets?

This would need to work in IE6+, Firefox and Chrome. Update: forgot to mention that (a) we are using Grails and (b) this needs to run over https.

Solution 1

You can try Plupload. It can be configured to check whatever runtime is available on users side, be it - Flash, Silverlight, HTML5, Gears, etc, and use whichever satisfies required features first. Among other things it supports image resizing (on users side, preserving EXIF data(!)), stream and multipart upload, and chunking. Files can be chunked on users side, and sent to a server-side handler chunk-by-chunk (requires some additional care on server), so that big files can be uploaded to a server having max filesize limit set to a value much lower then their size, for example. And more.

Some runtimes support https I believe, some need testing. Anyway, developers on there are quite responsive these days. So you might at least try ;)

Solution 2

The only option I know of that would allow this would be a signed Java applet.

Unsigned applets and Flash movies have no filesystem access, so they wouldn't be able to read the file data. Flash is able to upload files, but most of that is handled by the built-in Flash implementation and from what I remember the file contents would never be exposed to your code.

Solution 3

There is no JavaScript solution for that selection of browsers. There is the File API but whilst it works in newer Firefox and Chrome versions it's not going to happen in IE (no sign of it in IE9 betas yet either).

In any case, reading the file locally and uploading it via XMLHttpRequest is inefficient because XMLHttpRequest does not have the ability to send pure binary, only Unicode text. You can encode binary into text using base-64 (or, if you are really dedicated, a custom 7-bit encoding of your own) but this will be less efficient than a normal file upload.

You can certainly do uploads with Flash (see SWFUpload et al), or even Java if you must (Jumploader... I wouldn't bother, these days, though, as Flash prevalence is very high and the Java plugin continues to decline). You won't necessarily get the low-level control to split into chunks, but do you really need that? What for?

Another possible approach is to use a standard HTML file upload field, and when submit occurs set an interval call to poll the server with XMLHttpRequest, asking it how far the file upload is coming along. This requires a bit of work on the server end to store the current upload progress in the session or database, so another request can read it. It also means using a form parsing library that gives you progress callback, which most standard language built-in ones like PHP's don't.

Whatever you do, take a ‘progressive enhancement’ approach, allowing browsers with no support to fall back to a plain HTML upload. Browsers do typically have an upload progress bar for HTML file uploads, it just tends to be small and easily missed.

Comments

[Pekka]

This is probably possible in both Flash and Java, but why? What sense would this make? Do you want to bypass size limitations?

[RyanW]

Here's another similar question: stackoverflow.com/questions/3155587/…

[Yevgeniy Brikman]

I am mainly exploring this option to avoid having to handle huge files server side. The contents of the uploaded files will need to be parsed and stored in a DB. For a large amount of data at once, we'd need a dedicated pool of long lived threads on the front ends to transfer all the data, possibly first to a backend service via RPC and then the DB itself. Then there would be the issues of handling file cleanup, retries, frontend servers crashing, memory management and so on. All of this is doable, but if it could be done on the client side, I could get the solution out the door much faster.

[Chaki_Black]

Try use this: stackoverflow.com/a/12056417/551744 . I relialized splitting files with JavaScript and merging them on server side. This can help too: html5rocks.com/ru/tutorials/file/dndfiles

[MarioRicalde]

Flash can't do it. It has file upload capabilities, but the upload stuff is hidden from the client code. There's no access to the actual file data. Unsigned Java applets probably can't do it either. I'm fairly certain you would need to use a signed Java applet.

[Marcel Korpel]

Hmmm, that bar on the bottom side of IE (6 & 7) at work is very misleading: when I send large files (usually 10–50 MB) the bar is full way earlier than the file upload actually ends.

[Amit Patil]

Yeah. There are definitely problems with it especially in IE. It's a bit sad how there has been so much effort put into circumventing the HTML file upload control, when it could have been solved much more easily with a little work from browser vendors making their existing UI a bit better.

[Yevgeniy Brikman]

What is the frontend technology behind these "streaming upload components" like SlickUpload? Are they able to manipulate files before uploading?

[Lars Blåsjö]

From Flash Player 10, you can read file data using ActionScript: mikechambers.com/blog/2008/08/20/…

[MarioRicalde]

Interesting. I didn't realize they'd added that.

[EricP]

I've "successfully" implemented chunked file uploading in Flash using FileReference. I say "successfully" because flash must read the entire file into memory. Uploading a 700MB file will make flash unresponsive for 30 seconds before the upload even starts.

[jayarjo]

Does Flash really read whole file into memory, before upload? Never heard of that. Is there any doc around describing the issue?

[Tomas]

I'm not sure about your answer as at the plupload homepage you can read that the Flash version is capable of file chunking.