Running an AWS cli command from a local python script?

Question

I'm trying to run this aws s3 ls command:

aws s3 ls s3://path/to/my/bucket/12434 --recursive --human-readable --summarize

with this python:

command = 'aws s3 ls s3://path/to/my/bucket/12434 --recursive --human-readable --summarize'
s3_folder_data  = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True)
print s3_folder_data

But it's failing with this error:

subprocess.CalledProcessError: Command 'aws s3 ls s3://path/to/my/bucket/12434 --recursive --human-readable --summarize' returned non-zero exit status 1

The command itself works when I run it. The python script is being called by the same user on the same machine. What gives?

Solution 1

As suggested by others, use Boto3 S3 library to get what you want. But if you insist on subprocess, try:

subprocess.check_output(['aws', 's3', 'ls', 's3://path/to/my/bucket/12434', '--recursive', '--human-readable', '--summarize'])

or

subprocess.call(['aws', 's3', 'ls', 's3://path/to/my/bucket/12434', '--recursive', '--human-readable', '--summarize'])

and build on it.

Solution 2

New in Python 3.5, you can also use subprocess.run().

subprocess.run(['aws', 's3', 'ls', 's3://path/to/my/bucket/12434', '--recursive', '--human-readable', '--summarize'])