Folder won't delete on Amazon S3

Question

I'm trying to delete a folder created as a result of a MapReduce job. Other files in the bucket delete just fine, but this folder won't delete. When I try to delete it from the console, the progress bar next to its status just stays at 0. Have made multiple attempts, including with logout/login in between.

Solution 1

First and foremost, Amazon S3 doesn't actually have a native concept of folders/directories, rather is a flat storage architecture comprised of buckets and objects/keys only - the directory style presentation seen in most tools for S3 (including the AWS Management Console itself) is based solely on convention, i.e. simulating a hierarchy for objects with identical prefixes - see my answer to How to specify an object expiration prefix that doesn't match the directory? for more details on this architecture, including quotes/references from the AWS documentation.

Accordingly, your problem might stem from a tool using a different convention for simulating this hierarchy, see for example the following answers in the AWS forums:

• Ivan Moiseev's answer to the related question Cannot delete file from bucket, where he suggests to use another tool to inspect whether you might have such a problem and remedy it accordingly.

• The AWS team response to What are these _$folder$ objects? - This is a convention used by a number of tools including Hadoop to make directories in S3. They're primarily needed to designate empty directories. One might have preferred a more aesthetic scheme, but well that is the way that these tools do it.

Good luck!

Solution 2

I had the same issue and used AWS CLI to fix it:

aws s3 rm s3://<your-bucket>/<your-folder-to-delete>/ --recursive ;

(this assumes you have run aws configure and aws s3 ls s3://<your-bucket>/ already works)

Solution 3

I was getting the following error when I tried to delete a bucket which was a directory that held log files from Cloudfront.

An unexpected error has occurred. Please try again later.

After I disabled logging in Cloudfront I was able to delete the folder successfully.

My guess is that it was a system folder used by Cloudfront that did not allow deletion by the owner.

In your case, you may want to check if MapReduce is holding on to the folder in question.

Solution 4

I was facing the same problem. Tried many login, logout attempts and refresh but problem persist. Searched stackoverflow and found suggestions to cut and paste folder in different folder then delete but didn't worked. Another thing you should look is for versioning that might effect your bucket may be suspending the versioning allow you to delete the folder.

My solution was to delete it with code. I have used boto package in python for file handling over s3 and the deletion worked when I tried to delete that folder from my python code.

import boto
from boto.s3.key import Key
keyId = "your_aws_access_key"
sKeyId = "your_aws_secret_key"
fileKey="dummy/foldertodelete/" #Name of the file to be deleted
bucketName="mybucket001" #Name of the bucket, where the file resides
conn = boto.connect_s3(keyId,sKeyId) #Connect to S3
bucket = conn.get_bucket(bucketName) #Get the bucket object
k = Key(bucket,fileKey) #Get the key of the given object
k.delete() #Delete

S3 doesn't keep directory it just have a flat file structure so everything is managed with key. For you its a folder but for S3 it just an key.

If you want to delete a folder named -> dummy then key would be

fileKey = "/dummy/"

View more solutions

Comments

[AdjunctProfessorFalcon]

Works like a charm, definitely the way to go. And for anyone who's curious, if you want to upload a local directory, the opposite command is aws s3 cp YourLocalProjectFolder s3://yourS3bucketname --recursive --debug

[YuAn Shaolin Maculelê Lai]

There is no ';' in the end of line. aws> s3 rm s3://<your-bucket>/<your-folder-to-delete>/ --recursive

[sonjz]

@YuAnShaolinMaculelêLai ';' is optional in powershell or bash. I admit, we have entered a post-modern era of a semi-colonlessness (ruby/python)... but it has its uses: allows multiple commands on a single line, or denote the end of a statement.

[YuAn Shaolin Maculelê Lai]

@sonjz I got it. Thank you.

[Cyclonecode]

+1 - This worked fine for me when I had created a folder with the invalid name folder:, (notice the trailing colon) that I was not able to delete using cyberduck.

[Niklas Rosencrantz]

This did not work for me. I am checking why it won't work.

[sonjz]

@NiklasRosencrantz i'm guessing your IAM permissions, or perhaps your --region or --profile needs to be specified. aws s3 ls --region us-east-1 --profile default is a good test to see if you can connect (given you have run aws configure already any your permissions at AWS are set)