AWS CLI S3: copying file locally using the terminal : fatal error: An error occurred (404) when calling the HeadObject operation

Question

I'm trying to copy files locally from s3 bucket. I can get the list of files on my bucket:

aws s3 ls  s3://myBucket/myDirectory/todaysFiles/

But when I try to copy the files locally:

aws s3 cp s3://myBucket/myDirectory/todaysFiles/ .

I get this error:

fatal error: An error occurred (404) when calling the HeadObject operation: Key "myDirectory/todaysFiles/" does not exist

But I try to copy just one file locally:

 aws s3 cp s3://myBucket/myDirectory/todaysFiles/somefile .

I get this error:

 warning: Skipping file s3://myBucket/myDirectory/todaysFiles/somefile. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to the perform operation. See aws s3 download help for additional parameter options to ignore or force these transfers.

Any of you knows why I'm getting this error or way around this errors?

I really appreciate your help

Solution 1

For the first error - add the recursive flag:

aws s3 cp s3://myBucket/myDirectory/todaysFiles/ . --recursive

This will copy all the files in the "todaysFiles" directory to the current directory.

However, the second error indicates that your files are in Glacier. This complicates things a bit as Glacier is not real time - depending on what you're willing to pay it can be hours before the data is restored. See the Restoring Objects docs for a bit more information. You can't copy from S3 until the object are restored from Glacier to S3.

Note that if you do this you will have costs from both Glacier and S3.

As an aside, if these files are really files from today there should be a much longer time between storage on S3 and the push to Glacier. But I'm guessing that the parent directories may have a date related component too.

Solution 2

Use --recursive at the end if the Source contains multiple folders.

aws s3 cp s3Uri Local --recursive

Comments

[AlwaysLearning]

The --region switch applies to the destination bucket, which is meaningless here because it's the local file system. To specify the region of the source bucket you use the --source-region switch instead. REF: docs.aws.amazon.com/cli/latest/reference/s3/cp.html

[Luigi Lopez]

Looks like OP already had awscli installed, no need to suggest it.

[DJ_Stuffy_K]

s3 is global isn't it? what's the point of specifying the region?