AWS s3 listobjects with pagination

Question

I want to implement pagination using aws s3. There are 500 files in object ms.files but i want to retrieve only 20 files at a time and next 20 next time and so on.

var params = {
  Bucket: 'mystore.in',
  Delimiter: '/',
  Prefix: '/s/ms.files/',
  Marker:'images',
};
s3.listObjects(params, function(err, data) {
  if (err) console.log(err, err.stack); 
  else     console.log(data);          
});

Solution 1

Came across this while looking to list all of the objects at once, if your response is truncated it gives you a flag isTruncated = true and a continuationToken for the next call

If youre on es6 you could do this,

const AWS = require('aws-sdk');
const s3 = new AWS.S3({});

const listAllContents = async ({ Bucket, Prefix }) => {
  // repeatedly calling AWS list objects because it only returns 1000 objects
  let list = [];
  let shouldContinue = true;
  let nextContinuationToken = null;
  while (shouldContinue) {
    let res = await s3
      .listObjectsV2({
        Bucket,
        Prefix,
        ContinuationToken: nextContinuationToken || undefined,
      })
      .promise();
    list = [...list, ...res.Contents];

    if (!res.IsTruncated) {
      shouldContinue = false;
      nextContinuationToken = null;
    } else {
      nextContinuationToken = res.NextContinuationToken;
    }
  }
  return list;
};

Solution 2

Solution as shared by Mr Jarmod :

var params = {
  Bucket: 'mystore.in',
  Delimiter: '/',
  Prefix: '/s/ms.files/',
  Marker:'',
  MaxKeys : 20
};
s3.listObjects(params, function(err, data) {
  if (err) console.log(err, err.stack); 
  else     console.log(data);          
});

Comments

[MountainBiker]

First of all, I just used your example, so thank you! I have been doing some reading up on how to use the spread operator. From what I understand , the line " list = [...list, ...res.Contents];" is basically reading the existing ...list array on each iteration of the while loop and copying it AND the new results of the call to s3.listObjectsV2 (in this case ...res.Content)?

[David Cheung]

looking back at the implementation I feel like list.push(...res.Contents) would be better, as it does not declare a new array every loop. The line list = [] is declaring a new array, and ...list is appending each item of list into the array, same with ...res.Contents

[Ilya Kovalyov]

It doesn't have any info about pagination