How to Upload CSV file on S3 Bucket using NodeJS?

Question

I am creating a CSV file on the fly from JSON content and uploading generated CSV file on S3 bucket rather than first saving the file locally.

Below is my code snippet, as using below code my CSV file uploaded on S3 bucket but it does not seems to be in correct CSV format.

var uploadCSVFileOnS3Bucket = function(next, csvFileContent,results) {
    console.log("uploadCSVFileOnS3Bucket function started");
    var bufferObject = new Buffer.from(JSON.stringify(csvFileContent));
    var filePath = configurationHolder.config.s3UploadFilePath;
    var s3 = new AWS.S3();
    var params = {
        Bucket: 'bucket_name'
        Key: 's3UploadFilePath',
        Body: bufferObject,
        CacheControl:'public, max-age=86400'
    }
    s3.upload(params, function(err, data) {
        if (err) {
            console.log("Error at uploadCSVFileOnS3Bucket function",err);
            next(err);
        } else {
            console.log("File uploaded Successfully");
            next(null, filePath);
        }
    });
};

Also, I am using "json2csv" npm module for generating csv file content from JSON.

Below is the code:

var generateCSVFile = function(next,callback,csvFileContent) {
   console.log("generateCSVFile function started",csvFileContent);
   if(csvFileContent && csvFileContent.length>0) {
     var fields = ['field1','field2','field3',........];
     var csv = json2csv({ data: csvFileContent, fields: fields });
     console.log('created',csv);
     next(null,csv);
   }
   else {
     next(null,[]);
   }
 }

Please let us know where the above code is going wrong.

Solution 1

Hi I tried again with below headers values and it worked for me. Below is the code :

var s3 = new AWS.S3();
var params = {
    Bucket: bucketName,
    Key: filePath,
    Body: csvFileContent,
    ContentType: 'application/octet-stream',
    ContentDisposition: contentDisposition(filePath, {
        type: 'inline'
    }),
    CacheControl: 'public, max-age=86400'
}
s3.putObject(params, function(err, data) {
    if (err) {
        console.log("Error at uploadCSVFileOnS3Bucket function", err);
        next(err);
    } else {
        console.log("File uploaded Successfully");
        next(null, filePath);
    }
});

Solution 2

add ContentDisposition: 'attachment' in your params as well.

otherwise you can also read file and upload to s3

fs.readFile(FILEPATH, function(err, file_buffer) {
            var params = {
                Bucket:  //bucketname,
                Key:key,
                ContentDisposition: 'attachment',
                Body: file_buffer
            };
            s3.upload(params, function(err, data) {
                if (err) {
                    console.log("Error in upload");
                    callback(err, null)
                }
                if (data) {
                    console.log("Upload Success", data);
                    callback(null, data)
                }
            });
});

Comments

[Ankit Uniyal]

I tried using the above Header but it doesn't provided me the correct result.

[Ankit Uniyal]

Hi I tried again with the above header and it works, I added the whole code in a separate comment. Thanks

[Paulo Roberto Rosa]

and this contentDisposition funcion where it is??

[Arjun Singh]

CacheControl - does this option clear cache before uploading the file?

[Ankit Uniyal]

@ArjunSingh No, actually this CacheControl sets a cache time to the CDN when we try to access S3 object from its linked CDN URL. To remove cache, we manually need to remove the CDN cache else it will only point to S3 object once its cache time expires.