How to create a "folder-like" (i.e. PREFIX) object on S3 using the AWS CLI?

16,614

Solution 1

After some quick fiddling around I found this seems to work:

aws s3api put-object --bucket test --key dir-test/

But it only works if you include the "/" at the end of the argument to --key. That part just didn't feel right... surely there's a better way?

Solution 2

There really, really aren't directories in S3. However, as you point out, there are prefixes and delimiters and the default delimiter is / which allows you to get a pretty convincing simulation of a hierarchical directory structure in an S3 bucket.

But the bucket is still just a flat space containing objects with key names. If you want to create a directory you have to create an object with a key whose name includes or ends with a delimiter character (/ by default). So your technique described above may not feel right but it is the only way.

And whoever came up with the Create Folder idea in the console should be ashamed of themselves. It causes a lot of confusion.

Share:
16,614
Cawb07
Author by

Cawb07

I love to write code and solve problems & puzzles in my free time.

Updated on July 21, 2022

Comments

  • Cawb07
    Cawb07 over 1 year

    It is usually explained that there are no folders/directories on S3, which is true, however there are PREFIX objects. In some cases - e.g. using riofs to "mount" a filesystem to S3 - it could actually be useful to know how to create one of these.

    Does anyone know if there is a "correct" way to do this with the AWS CLI? It's likely possible using the low-level API aws s3api ...

    Related SO posts:

    Amazon S3 boto - how to create a folder?

    Creating a folder via s3cmd (Amazon S3)

    P.S.

    I also want to point out that in the AWS console this action is actually named "Create Folder...", so it's not really fair to tell people that there is no "concept" of a folder on S3.

    Many thanks!

    • greg_diesel
      greg_diesel about 8 years
      i've used s3fuse in the past to 'mount' s3 as a filesystem. I thought that when you did a mkdir /var/mys3mount/newfolder/ it only created the folder in the local file system, and not in S3. It was giving the appearance of a folder existing but s3 wouldn't see it until you put an object there. Then giving you the full path in the folder concept.
    • Cawb07
      Cawb07 about 8 years
      Hi Greg, I've seen the same behavior with riofs when using mkdir locally, but I've also seen that when a prefix object exists on the S3 bucket, riofs will create a local directory to mirror this. This can be useful when automating things around these "mounts".
  • Davos
    Davos over 6 years
    Technically that's true, but most customers could care less what the underlying storage mechanism is, that it's not a file system. s3 is a kind of URI scheme, and those were originally created as abstractions over hierarchical sub-directories. Who cares that these are not really folders / directories. They look like folders, they smell like folders. It's just being pedantic to expect customers to not fall asleep when explaining that they aren't folders.