Boto3, s3 folder not getting deleted

Question

I have a directory in my s3 bucket 'test', I want to delete this directory. This is what I'm doing

s3 = boto3.resource('s3')
s3.Object(S3Bucket,'test').delete()

and getting response like this

{'ResponseMetadata': {'HTTPStatusCode': 204, 'HostId': '************', 'RequestId': '**********'}}

but my directory is not getting deleted!

I tried with all combinations of '/test', 'test/' and '/test/' etc, also with a file inside that directory and with empty directory and all failed to delete 'test'.

Solution 1

delete_objects enables you to delete multiple objects from a bucket using a single HTTP request. You may specify up to 1000 keys.

https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.delete_objects

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')

objects_to_delete = []
for obj in bucket.objects.filter(Prefix='test/'):
    objects_to_delete.append({'Key': obj.key})

bucket.delete_objects(
    Delete={
        'Objects': objects_to_delete
    }
)

Solution 2

NOTE: See Daniel Levinson's answer for a more efficient way of deleting multiple objects.

---

In S3, there are no directories, only keys. If a key name contains a / such as prefix/my-key.txt, then the AWS console groups all the keys that share this prefix together for convenience.

To delete a "directory", you would have to find all the keys that whose names start with the directory name and delete each one individually. Fortunately, boto3 provides a filter function to return only the keys that start with a certain string. So you can do something like this:

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket-name')
for obj in bucket.objects.filter(Prefix='test/'):
    s3.Object(bucket.name, obj.key).delete()

Comments

[baldr]

This should be accepted answer as it is better than from David Morales

[colllin]

objects_to_delete = [{'Key': obj.key} for obj in bucket.objects.filter(Prefix='test', Marker='test/path/to/dir')]

[y2k-shubham]

This extension of the same approach appears to be more comprehensive