How to display only files from aws s3 ls command?
Solution 1
You can't do this with just the aws
command, but you can easily pipe it to another command to strip out the portion you don't want. You also need to remove the --human-readable
flag to get output easier to work with, and the --summarize
flag to remove the summary data at the end.
Try this:
aws s3 ls s3://mybucket --recursive | awk '{print $4}'
Edit: to take spaces in filenames into account:
aws s3 ls s3://mybucket --recursive | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ \t]*//'
Solution 2
Use the s3api with jq (AWS docu aws s3api list-objects):
This mode is always recursive.
$ aws s3api list-objects --bucket "bucket" | jq -r '.Contents[].Key'
a.txt
foo.zip
foo/bar/.baz/a
[...]
You can filter sub directories by adding a prefix (here foo
directory). The prefix must not start with an /
.
$ aws s3api list-objects --bucket "bucket" --prefix "foo/" | jq -r '.Contents[].Key'
foo/bar/.baz/a
foo/bar/.baz/b
foo/bar/.baz/c
[...]
jq Options:
-
-r
= Raw Mode, no quotes in output -
.Contents[]
= GetContents
Object Array Content -
.Key
= Get every Key Field (does not produce a valid JSON Array, but we are in raw mode, so we don't care)
Addendum:
You can use pure AWS CLI, but the values will be seperated by \x09
= Horizontal Tab (AWS: Controlling Command Output from the AWS CLI - Text Output Format)
$ aws s3api list-objects --bucket "bucket" --prefix "foo/" --query "Contents[].Key" --output text
foo/bar/.baz/a foo/bar/.baz/b foo/bar/.baz/c [...]
AWS CLI Options:
-
--query "Contents[].Key"
= Query Contents Object Array and get every Key inside -
--output text
= Output as Tab delimited Text with now Quotes
Addendum based on Guangyang Li Comment:
Pure AWS CLI with New Line:
$ aws s3api list-objects --bucket "bucket" --prefix "foo/" --query "Contents[].{Key: Key}" --output text
foo/bar/.baz/a
foo/bar/.baz/b
foo/bar/.baz/c
[...]
Solution 3
A simple filter would be:
aws s3 ls s3://mybucket --recursive | perl -pe 's/^(?:\S+\s+){3}//'
This will remove the date, time and size. Left only the full path of the file. It also works without the recursive and it should also works with filename containing spaces.
Solution 4
Simple Way
aws s3 ls s3://mybucket --recursive --human-readable --summarize|cut -c 29-
Solution 5
My Solution
List only files recursively using aws cli.
aws s3 ls s3://myBucket --recursive | awk 'NF>1{print $4}' | grep .
grep .
- Clear empty lines.
Example: aws s3 ls s3://myBucket
PRE f5c10c1678e8484482964b8fdcfe43ad/
PRE f65b94ad31734135a61a7fb932f7054d/
PRE f79b12a226b542dbb373c502bf125ffb/
PRE logos/
PRE test/
PRE userpics/
2019-05-14 10:56:28 7754 stage.js
Solution: aws s3 ls s3://myBucket --recursive | awk 'NF>1{print $4}' | grep .
stage.js
Borealis
Updated on July 08, 2022Comments
-
Borealis almost 2 years
I am using the aws cli to list the files in an s3 bucket using the following command (documentation):
aws s3 ls s3://mybucket --recursive --human-readable --summarize
This command gives me the following output:
2013-09-02 21:37:53 10 Bytes a.txt 2013-09-02 21:37:53 2.9 MiB foo.zip 2013-09-02 21:32:57 23 Bytes foo/bar/.baz/a 2013-09-02 21:32:58 41 Bytes foo/bar/.baz/b 2013-09-02 21:32:57 281 Bytes foo/bar/.baz/c 2013-09-02 21:32:57 73 Bytes foo/bar/.baz/d 2013-09-02 21:32:57 452 Bytes foo/bar/.baz/e 2013-09-02 21:32:57 896 Bytes foo/bar/.baz/hooks/bar 2013-09-02 21:32:57 189 Bytes foo/bar/.baz/hooks/foo 2013-09-02 21:32:57 398 Bytes z.txt Total Objects: 10 Total Size: 2.9 MiB
However, this is my desired output:
a.txt foo.zip foo/bar/.baz/a foo/bar/.baz/b foo/bar/.baz/c foo/bar/.baz/d foo/bar/.baz/e foo/bar/.baz/hooks/bar foo/bar/.baz/hooks/foo z.txt
How can I omit the date, time and file size in order to show only the file list?
-
Michal Gasek about 8 yearsPartially correct answer - it will not work if there's a whitespace in filename. Also
awk '{print $5}'
, not$4
-
Mark B about 8 years@MichalGasek if you remove the
--human-readable
flag like I specified, then it's $4, not $5. -
Michal Gasek about 8 yearsRight, it's $4 then. It still should deal with whitespaces in filenames... otherwise surprises come up.
-
Mark B about 8 years@MichalGasek I invite you to post an answer that deals with spaces in filenames.
-
Michal Gasek about 8 yearsDon't think it's worth another answer really. Piping through perl and matching after 3rd whitespace could for example work fine here:
aws s3 ls s3://mybucket --recursive | perl -ne '($key)=$_=~/^[\d\-]+\s+[\d\:]+\s+\d+\s(.+?)$/g; print "$key\n";'
-
LateralFractal over 7 yearsAlternate non-awk solution:
aws s3 ls s3://mybucket --recursive | tr -s ' ' | cut -d' ' -f4
-
michael over 6 yearscurrently, for me,
aws s3 ls
outputs such that you'd want to cut on-c32
, not-c29
; not sure if it's my data or a change in output format. (I don't actually have subfolders.) This is true for--human-readable
or plain default output; columns are the same place. But really, there's no need for human-readable in this case. And in either case you'd want to omit the--summarize
. In short,aws s3 ls s3://mybucket | cut -c32-
(and--recursive
only if desired) -
michael over 6 yearsNote that all the other answers here that attempt to cut based on spaces (awk, cut, whatever) are not going to work if there are spaces in the filenames.
-
michael over 6 yearsI can't verify if this works for recursive, but since the "simple" version won't work for spaces in filename, it seems like a fragile solution, and the other is needlessly complex. Instead, cut on chars, which should be robust enough until the CLI output format changes:
aws s3 ls s3://mybucket | cut -c32-
(optionally add recursive & verify it still works) -
Josiah over 5 yearsThis doesn't seem to work as consistently as the awk solution for me.
-
f_i about 5 yearsvery nice. or
aws s3api list-buckets | jq -r '.Buckets[].Name'
-
brainstorm almost 5 yearsBreaks with --human-readable... edit: and without too, actually.
-
Guangyang Li over 4 yearsI like the pure AWS CLI one and actually you can do it with
--query 'Contents[].{Key: Key}'
. Then it will be one record per line. -
Yannick Wurm about 4 yearsThis is the cleanest way to do it (as for michael with -c32)
-
LonelyRogue about 4 yearsdoesn't work when file name has spaces like "ChatNotes 8Mar.txt". This code prints only "ChatNotes"
-
Ste over 3 yearsNot all heroes wear capes
-
Shivam Singh about 3 yearsThanks. This works perfectly with spaces and even with tabs in the filename. Saves a lot and time and effort.
-
MultiDev over 2 yearsGreat concept, but does not work when there are spaces in the file name.
-
Shoukry over 2 yearsTrue... This should work then:
aws s3api list-objects --bucket mybucket --prefix myprefix --query 'Contents[].Key' | jq -rc '.[]'
-
Rahul Mandaliya about 2 yearsThis is solution without use of s3api