How do I delete a versioned bucket in AWS S3 using the CLI?

Question

I have tried both s3cmd:

$ s3cmd -r -f -v del s3://my-versioned-bucket/

And the AWS CLI:

$ aws s3 rm s3://my-versioned-bucket/ --recursive

But both of these commands simply add DELETE markers to S3. The command for removing a bucket also doesn't work (from the AWS CLI):

$ aws s3 rb s3://my-versioned-bucket/ --force
Cleaning up. Please wait...
Completed 1 part(s) with ... file(s) remaining
remove_bucket failed: s3://my-versioned-bucket/ A client error (BucketNotEmpty) occurred when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

Ok... how? There's no information in their documentation for this. S3Cmd says it's a 'fully-featured' S3 command-line tool, but it makes no reference to versions other than its own. Is there any way to do this without using the web interface, which will take forever and requires me to keep my laptop on?

Solution 1

One way to do it is iterate through the versions and delete them. A bit tricky on the CLI, but as you mentioned Java, that would be more straightforward:

AmazonS3Client s3 = new AmazonS3Client();
String bucketName = "deleteversions-"+UUID.randomUUID();

//Creates Bucket
s3.createBucket(bucketName);

//Enable Versioning
BucketVersioningConfiguration configuration = new BucketVersioningConfiguration(ENABLED);
s3.setBucketVersioningConfiguration(new SetBucketVersioningConfigurationRequest(bucketName, configuration ));

//Puts versions
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("some-bytes".getBytes()), null);
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("other-bytes".getBytes()), null);

//Removes all versions
for ( S3VersionSummary version : S3Versions.inBucket(s3, bucketName) ) {
    String key = version.getKey();
    String versionId = version.getVersionId();          
    s3.deleteVersion(bucketName, key, versionId);
}

//Removes the bucket
s3.deleteBucket(bucketName);
System.out.println("Done!");

You can also batch delete calls for efficiency if needed.

Solution 2

I ran into the same limitation of the AWS CLI. I found the easiest solution to be to use Python and boto3:

#!/usr/bin/env python

BUCKET = 'your-bucket-here'

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET)
bucket.object_versions.delete()

# if you want to delete the now-empty bucket as well, uncomment this line:
#bucket.delete()

A previous version of this answer used boto but that solution had performance issues with large numbers of keys as Chuckles pointed out.

Solution 3

Using boto3 it's even easier than with the proposed boto solution to delete all object versions in an S3 bucket:

#!/usr/bin/env python
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
bucket.object_versions.all().delete()

Works fine also for very large amounts of object versions, although it might take some time in that case.

Solution 4

You can delete all the objects in the versioned s3 bucket. But I don't know how to delete specific objects.

$ aws s3api delete-objects \
      --bucket <value> \
      --delete "$(aws s3api list-object-versions \
      --bucket <value> | \
      jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

Alternatively without jq:

$ aws s3api delete-objects \
    --bucket ${bucket_name} \
    --delete "$(aws s3api list-object-versions \
    --bucket "${bucket_name}" \
    --output=json \
    --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Solution 5

This two bash lines are enough for me to enable the bucket deletion !

1: Delete objects aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

2: Delete markers aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

View more solutions