How to upload a file from the browser to Amazon S3 with node.js, Express, and knox?

Question

I'm trying to find some example code that utilizes node.js, Express, and knox.

The docs for Knox only give clear examples of how to upload a file already stored in the file system. https://github.com/learnboost/knox#readme

Additionally, there a number of simple tutorials (even in Express itself) on how to upload files directly to express and save to the file system.

What I'm having trouble finding is an example that lets you upload a client upload to a node server and have the data streamed directly to S3 rather than storing in the local file system first.

Can someone point me to a gist or other example that contains this kind of information?

Solution 1

All of the previous answers involve having the upload pass through your node.js server which is inefficient and unnecessary. Your node server does not have to handle the bandwidth or processing of uploaded files whatsoever because Amazon S3 allows uploads direct from the browser.

Have a look at this blog post: http://blog.tcs.de/post-file-to-s3-using-node/

I have not tried the code listed there, but having looked over it, it appears solid and I will be attempting an implementation of it shortly ad will update this answer with my findings.

Solution 2

Here is an example of streaming directly to s3 without ever touching your hard drive, using multiparty and knox:

var http = require('http')
  , util = require('util')
  , multiparty = require('multiparty')
  , knox = require('knox')
  , Batch = require('batch')
  , PORT = process.env.PORT || 27372

var s3Client = knox.createClient({
  secure: false,
  key: process.env.S3_KEY,
  secret: process.env.S3_SECRET,
  bucket: process.env.S3_BUCKET,
});

var Writable = require('readable-stream').Writable;
util.inherits(ByteCounter, Writable);
function ByteCounter(options) {
  Writable.call(this, options);
  this.bytes = 0;
}

ByteCounter.prototype._write = function(chunk, encoding, cb) {
  this.bytes += chunk.length;
  cb();
};

var server = http.createServer(function(req, res) {
  if (req.url === '/') {
    res.writeHead(200, {'content-type': 'text/html'});
    res.end(
      '<form action="/upload" enctype="multipart/form-data" method="post">'+
      '<input type="text" name="path"><br>'+
      '<input type="file" name="upload"><br>'+
      '<input type="submit" value="Upload">'+
      '</form>'
    );
  } else if (req.url === '/upload') {
    var headers = {
      'x-amz-acl': 'public-read',
    };
    var form = new multiparty.Form();
    var batch = new Batch();
    batch.push(function(cb) {
      form.on('field', function(name, value) {
        if (name === 'path') {
          var destPath = value;
          if (destPath[0] !== '/') destPath = '/' + destPath;
          cb(null, destPath);
        }
      });
    });
    batch.push(function(cb) {
      form.on('part', function(part) {
        if (! part.filename) return;
        cb(null, part);
      });
    });
    batch.end(function(err, results) {
      if (err) throw err;
      form.removeListener('close', onEnd);
      var destPath = results[0]
        , part = results[1];

      var counter = new ByteCounter();
      part.pipe(counter); // need this until knox upgrades to streams2
      headers['Content-Length'] = part.byteCount;
      s3Client.putStream(part, destPath, headers, function(err, s3Response) {
        if (err) throw err;
        res.statusCode = s3Response.statusCode;
        s3Response.pipe(res);
        console.log("https://s3.amazonaws.com/" + process.env.S3_BUCKET + destPath);
      });
      part.on('end', function() {
        console.log("part end");
        console.log("size", counter.bytes);
      });
    });
    form.on('close', onEnd);
    form.parse(req);

  } else {
    res.writeHead(404, {'content-type': 'text/plain'});
    res.end('404');
  }

  function onEnd() {
    throw new Error("no uploaded file");
  }
});
server.listen(PORT, function() {
  console.info('listening on http://0.0.0.0:'+PORT+'/');
});

example taken from https://github.com/superjoe30/node-multiparty/blob/master/examples/s3.js

Solution 3

the node/express code doesn't work with nodejs v0.4.7

here is the updated code for nodejs v0.4.7

app.post('/upload', function (req, res) {
  // connect-form additions
  req.form.complete(function (err, fields, files) {
    // here lies your uploaded file:
    var path = files['upload-file']['path'];
    // do knox stuff here
  });
});

Comments

[Philipp Kyeck]

"All of the previous answers involve having the upload pass through your node.js server which is inefficient and unnecessary." but at the time of my answer (2011) your solution wasn't available. but i agree: now, it's the best way.

[balupton]

Interesting! What are the security precautions we have to put into place when considering client side uploading directly to S3? I imagine we wouldn't want to give anyone and everyone that ability. Thanks :)

[Michael Galaxy]

I'm using an s3 "compatible" service (vBlob from cloudfoundry) which doesn't accept method="post" but only method="put". Any ideas?

[coiso]

isn't it unsafe having amazon codes in the client side?

[kentcdodds]

Thanks for this, would love your help on my question here: stackoverflow.com/q/21873561/971592

[Alexander Zanfir]

your amazon/s3 codes are not on the client side, they are kept server side (where the node end point comes in), and are encoded and served to the client before the upload request to s3 is made

[OMGPOP]

what if attacker use the amazon id in the client code and use up your storage?

[talentedmrjones]

@OMGPOP The AWS access key (id) is essentially a "public" key. It is meant to be used in such cases. In order for someone to make use of the access key they would also need the secret key which is never sent out to the public. The form in this example would send the AWS access key and signature which is calculated from the secret key on the backend and delivered to the client

[nsp]

@talentedmrjones I create the s3Credentials on the server using secret key and passing the access key to the browser code. What if I am an adversary and I do the following: - I intercept your client to s3 request. - I change the s3 policy as it is available in plain text. - create a secret key and private key of my own. - create the s3Credentials again with the new data. - As the aws doesn't store those two ( awsSecretKey and AwsAccessKey) values it has to rely on whatever the client(now an adversary ) is sending. Now I can upload file of any size. Am I correct?

[nsp]

talentedmrjones has given a solution which is prone to man in the middle attack. There is one more solution which is given here terlici.com/2015/05/23/uploading-files-s3.html which is correct. But it has limitations like you cannot mention the file size etc.

[nsp]

@andrewrk In the above mentioned answer how should I stop the file upload before it completes. example : the file is currently uploading and I realise that the file is too big so I want to stop the upload by sending appropriatemessage. My node server hangs as I am not sending the correct response back when I want the upload to stop. stackoverflow.com/questions/33452952/…

[talentedmrjones]

@nsp Nope. Any change to any parameter of the request will invalidate the signature. When all else fails, read the documentation docs.aws.amazon.com/AmazonS3/latest/API/…