How to get response from S3 getObject in Node.js?
Solution 1
When doing a getObject()
from the S3 API, per the docs the contents of your file are located in the Body
property, which you can see from your sample output. You should have code that looks something like the following
const aws = require('aws-sdk');
const s3 = new aws.S3(); // Pass in opts to S3 if necessary
var getParams = {
Bucket: 'abc', // your bucket name,
Key: 'abc.txt' // path to the object you're looking for
}
s3.getObject(getParams, function(err, data) {
// Handle any error and exit
if (err)
return err;
// No error happened
// Convert Body from a Buffer to a String
let objectData = data.Body.toString('utf-8'); // Use the encoding necessary
});
You may not need to create a new buffer from the data.Body
object but if you need you can use the sample above to achieve that.
@aws-sdk/client-s3 (2021 Update)
Since I wrote this answer in 2016, Amazon has released a new JavaScript SDK, @aws-sdk/client-s3
. This new version improves on the original getObject()
by returning a promise always instead of opting in via .promise()
being chained to getObject()
. In addition to that, response.Body
is no longer a Buffer
but, one of Readable|ReadableStream|Blob
. This changes the handling of the response.Data
a bit. This should be more performant since we can stream the data returned instead of holding all of the contents in memory, with the trade-off being that it is a bit more verbose to implement.
In the below example the response.Body
data will be streamed into an array and then returned as a string. This is the equivalent example of my original answer. Alternatively, the response.Body
could use stream.Readable.pipe()
to an HTTP Response, a File or any other type of stream.Writeable
for further usage, this would be the more performant way when getting large objects.
If you wanted to use a Buffer
, like the original getObject()
response, this can be done by wrapping responseDataChunks
in a Buffer.concat()
instead of using Array#join()
, this would be useful when interacting with binary data. To note, since Array#join()
returns a string, each Buffer
instance in responseDataChunks
will have Buffer.toString()
called implicitly and the default encoding of utf8
will be used.
const { GetObjectCommand, S3Client } = require('@aws-sdk/client-s3')
const client = new S3Client() // Pass in opts to S3 if necessary
function getObject (Bucket, Key) {
return new Promise(async (resolve, reject) => {
const getObjectCommand = new GetObjectCommand({ Bucket, Key })
try {
const response = await client.send(getObjectCommand)
// Store all of data chunks returned from the response data stream
// into an array then use Array#join() to use the returned contents as a String
let responseDataChunks = []
// Handle an error while streaming the response body
response.Body.once('error', err => reject(err))
// Attach a 'data' listener to add the chunks of data to our array
// Each chunk is a Buffer instance
response.Body.on('data', chunk => responseDataChunks.push(chunk))
// Once the stream has no more data, join the chunks into a string and return the string
response.Body.once('end', () => resolve(responseDataChunks.join('')))
} catch (err) {
// Handle the error or throw
return reject(err)
}
})
}
@aws-sdk/client-s3
Documentation Links
Solution 2
Based on the answer by @peteb, but using Promises
and Async/Await
:
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
async function getObject (bucket, objectKey) {
try {
const params = {
Bucket: bucket,
Key: objectKey
}
const data = await s3.getObject(params).promise();
return data.Body.toString('utf-8');
} catch (e) {
throw new Error(`Could not retrieve file from S3: ${e.message}`)
}
}
// To retrieve you need to use `await getObject()` or `getObject().then()`
const myObject = await getObject('my-bucket', 'path/to/the/object.txt');
Solution 3
For someone looking for a NEST JS TYPESCRIPT
version of the above:
/**
* to fetch a signed URL of a file
* @param key key of the file to be fetched
* @param bucket name of the bucket containing the file
*/
public getFileUrl(key: string, bucket?: string): Promise<string> {
var scopeBucket: string = bucket ? bucket : this.defaultBucket;
var params: any = {
Bucket: scopeBucket,
Key: key,
Expires: signatureTimeout // const value: 30
};
return this.account.getSignedUrlPromise(getSignedUrlObject, params);
}
/**
* to get the downloadable file buffer of the file
* @param key key of the file to be fetched
* @param bucket name of the bucket containing the file
*/
public async getFileBuffer(key: string, bucket?: string): Promise<Buffer> {
var scopeBucket: string = bucket ? bucket : this.defaultBucket;
var params: GetObjectRequest = {
Bucket: scopeBucket,
Key: key
};
var fileObject: GetObjectOutput = await this.account.getObject(params).promise();
return Buffer.from(fileObject.Body.toString());
}
/**
* to upload a file stream onto AWS S3
* @param stream file buffer to be uploaded
* @param key key of the file to be uploaded
* @param bucket name of the bucket
*/
public async saveFile(file: Buffer, key: string, bucket?: string): Promise<any> {
var scopeBucket: string = bucket ? bucket : this.defaultBucket;
var params: any = {
Body: file,
Bucket: scopeBucket,
Key: key,
ACL: 'private'
};
var uploaded: any = await this.account.upload(params).promise();
if (uploaded && uploaded.Location && uploaded.Bucket === scopeBucket && uploaded.Key === key)
return uploaded;
else {
throw new HttpException("Error occurred while uploading a file stream", HttpStatus.BAD_REQUEST);
}
}
Solution 4
Updated (2022)
nodejs v17.5.0 added Readable.toArray. If this API is available in your node version. The code will be very short:
const buffer = Buffer.concat(
await (
await s3Client
.send(new GetObjectCommand({
Key: '<key>',
Bucket: '<bucket>',
}))
).Body.toArray()
)
If you are using Typescript, you are safe to cast the .Body
part as Readable
(the other types ReadableStream
and Blob
are only returned in browser environment. Moreover, in browser, Blob
is only used in legacy fetch API when response.body
is not supported)
(response.Body as Readable).toArray()
Note that: Readable.toArray
is an experimental (yet handy) feature, use it with caution.
=============
Original answer
If you are using aws sdk v3, the sdk v3 returns nodejs Readable (precisely, IncomingMessage which extends Readable) instead of a Buffer.
Here is a Typescript version. Note that this is for node only, if you send the request from browser, check the longer answer in the blog post mentioned below.
import {GetObjectCommand, S3Client} from '@aws-sdk/client-s3'
import type {Readable} from 'stream'
const s3Client = new S3Client({
apiVersion: '2006-03-01',
region: 'us-west-2',
credentials: {
accessKeyId: '<access key>',
secretAccessKey: '<access secret>',
}
})
const response = await s3Client
.send(new GetObjectCommand({
Key: '<key>',
Bucket: '<bucket>',
}))
const stream = response.Body as Readable
return new Promise<Buffer>((resolve, reject) => {
const chunks: Buffer[] = []
stream.on('data', chunk => chunks.push(chunk))
stream.once('end', () => resolve(Buffer.concat(chunks)))
stream.once('error', reject)
})
// if readable.toArray() is support
// return Buffer.concat(await stream.toArray())
Why do we have to cast response.Body as Readable
? The answer is too long. Interested readers can find more information on my blog post.
Solution 5
Extremely similar answer to @ArianAcosta above. Except I'm using import
(for Node 12.x and up), adding AWS config and sniffing for an image payload and applying base64
processing to the return
.
// using v2.x of aws-sdk
import aws from 'aws-sdk'
aws.config.update({
accessKeyId: process.env.YOUR_AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.YOUR_AWS_SECRET_ACCESS_KEY,
region: "us-east-1" // or whatever
})
const s3 = new aws.S3();
/**
* getS3Object()
*
* @param { string } bucket - the name of your bucket
* @param { string } objectKey - object you are trying to retrieve
* @returns { string } - data, formatted
*/
export async function getS3Object (bucket, objectKey) {
try {
const params = {
Bucket: bucket,
Key: objectKey
}
const data = await s3.getObject(params).promise();
// Check for image payload and formats appropriately
if( data.ContentType === 'image/jpeg' ) {
return data.Body.toString('base64');
} else {
return data.Body.toString('utf-8');
}
} catch (e) {
throw new Error(`Could not retrieve file from S3: ${e.message}`)
}
}
Related videos on Youtube
Sara Fuerst
I am the Founder and Lead Developer of Tibsar Software LLC. I have always had a passion for software development. Today I use this passion to help aspiring business owners build their software based business. I focus on creating quality software products that can scale with their business and meet their unique needs.
Updated on July 08, 2022Comments
-
Sara Fuerst almost 2 years
In a Node.js project I am attempting to get data back from S3.
When I use
getSignedURL
, everything works:aws.getSignedUrl('getObject', params, function(err, url){ console.log(url); });
My params are:
var params = { Bucket: "test-aws-imagery", Key: "TILES/Level4/A3_B3_C2/A5_B67_C59_Tiles.par"
If I take the URL output to the console and paste it in a web browser, it downloads the file I need.
However, if I try to use
getObject
I get all sorts of odd behavior. I believe I am just using it incorrectly. This is what I've tried:aws.getObject(params, function(err, data){ console.log(data); console.log(err); });
Outputs:
{ AcceptRanges: 'bytes', LastModified: 'Wed, 06 Apr 2016 20:04:02 GMT', ContentLength: '1602862', ETag: '9826l1e5725fbd52l88ge3f5v0c123a4"', ContentType: 'application/octet-stream', Metadata: {}, Body: <Buffer 01 00 00 00 ... > } null
So it appears that this is working properly. However, when I put a breakpoint on one of the
console.log
s, my IDE (NetBeans) throws an error and refuses to show the value of data. While this could just be the IDE, I decided to try other ways to usegetObject
.aws.getObject(params).on('httpData', function(chunk){ console.log(chunk); }).on('httpDone', function(data){ console.log(data); });
This does not output anything. Putting a breakpoint in shows that the code never reaches either of the
console.log
s. I also tried:aws.getObject(params).on('success', function(data){ console.log(data); });
However, this also does not output anything and placing a breakpoint shows that the
console.log
is never reached.What am I doing wrong?
-
peteb about 8 yearsIs your
aws
object actually a new instance of theaws.S3
object? Also, is the response fromgetObject()
being passed back to a http response or is it is being piped to a file? -
Sara Fuerst about 8 years@peteb
aws = new AWS.S3()
. The response should not be piped to a file. I need to use it in the Javascript -
peteb about 8 yearsSo then is it safe to assume that the contents are JSON or XML?
-
Sara Fuerst about 8 years@peteb neither, they are a custom file format
-
Mark B about 8 yearsShow the params you are using in the
getObject()
call. If you are trying to pass a signed URL to getObject I don't think that will work. -
Sara Fuerst about 8 years@peteb editted to show params
-
-
Sara Fuerst about 8 yearsSo the data coming back does seem to be a
Buffer
object, which I'm not familiar with. Theoretically I could usenew Buffer(data.Body).toString('utf-8');
to get to the content? -
peteb about 8 yearsIf the content is already a Buffer, no need to create a new Buffer from that. Simply just do
data.Body.toString('utf-8');
. A Buffer is a representation of Binary data in node, if you need more info here are the docs -
Sara Fuerst about 8 yearsThank you! This would be so much easier to work through if I could actually get a breakpoint working. For whatever reason (and this is the only place I've had this happen), within the function Netbeans refuses to show any variable values
-
carter about 6 yearsThis works for text, but is there a generic solution for handling text files as well as .png, .jpg, etc.?
-
peteb about 6 years@carter This is a general solution. Just change the
.toString('utf8')
when accessingdata.Body
to.toString('binary')
if you want a binary string for images. If theBuffer
indata.Body
doesn't need to be converted to a String like in this question, then you can just returndata.Body
and work with theBuffer
directly. -
Andrew Harris over 5 yearsThe .promise() on the end of getObject() was the key for me. I find the AWS SDK a little unintuitive at times.
-
jonask over 5 yearsMy response is saying 'Promise { <pending> }'
-
Arian Acosta over 5 years@jonask
getObject()
is an async function, did you try calling it withawait getObject(...)
? -
Dibish about 5 yearsTried mino, but how to get buffer data, when I print dataStream.Body its giving 'undefined'. ie console.log('datastream', dataStream.Body); //undefined
-
osullic almost 4 years"Convert Body from a Buffer to a String"... would be great if the AWS docs made this a bit more clear. I'm getting pretty fed up wrestling with AWS.
-
Ethan Standel over 2 yearsIt is absolutely insane that I can't find any documentation within AWS docs or on their NPM repo that just has this code. Like, did they really have to make it that hard? They couldn't just have a "for dummies" method (or even a stupid command class) that just gets the file contents b64 encoded or something with the rest of the object data? Like it grabs an absurd amount of garbage metadata that I, and I imagine most people, will NEVER NEED. But it can't return the object data itself?
-
Mohamed Iqzas over 2 yearsfileObject.Body returns either Buffer or undefined, and not able to return it using typescript
-
SSF over 2 years@peteb, your 2021 Update saved me after two days of being stuck on this exact problem. I ended up using a Buffer.concat() as per your suggestion to get a Buffer (which I later displayed as a Base64 image following this post: stackoverflow.com/questions/57699628/…). Thanks!
-
Divyanshu Juneja about 2 years@peteb I am getting some random set of symbols from the
responseDataChunks
. Any pointers for me?