Download file from AWS S3 using Python

Question

I am trying to download a file from Amazon S3 bucket to my local using the below code but I get an error saying "Unable to locate credentials"

Given below is the code I have written:

from boto3.session import Session
import boto3

ACCESS_KEY = 'ABC'
SECRET_KEY = 'XYZ'

session = Session(aws_access_key_id=ACCESS_KEY,
              aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('bucket_name')

for s3_file in your_bucket.objects.all():
    print(s3_file.key) # prints the contents of bucket

s3 = boto3.client ('s3')

s3.download_file('your_bucket','k.png','/Users/username/Desktop/k.png')

Could anyone help me on this. Thanks.

Solution 1

You are not using the session you created to download the file, you're using s3 client you created. If you want to use the client you need to specify credentials.

your_bucket.download_file('k.png', '/Users/username/Desktop/k.png')

or

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
s3.download_file('your_bucket','k.png','/Users/username/Desktop/k.png')

Solution 2

From an example in the official documentation, the correct format is:

import boto3

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')

You can also use a file-like object opened in binary mode.

s3 = boto3.client('s3', aws_access_key_id=... , aws_secret_access_key=...)
with open('FILE_NAME', 'wb') as f:
    s3.download_fileobj('BUCKET_NAME', 'OBJECT_NAME', f)
    f.seek(0)

---

The code in question uses s3 = boto3.client ('s3'), which does not provide any credentials.

The format for authenticating a client is shown here:

import boto3
client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

# Or via the Session
session = boto3.Session(
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

---

And lastly you can also re-use the authenticated session you created to get the bucket, and then download then file from the bucket.

from boto3.session import Session
import boto3

ACCESS_KEY = 'ABC'
SECRET_KEY = 'XYZ'

session = Session(aws_access_key_id=ACCESS_KEY,
              aws_secret_access_key=SECRET_KEY)

# session is authenticated and can access the resource in question 
session.resource('s3')
    .Bucket('bucket_name')
    .download_file('k.png','/Users/username/Desktop/k.png')

Solution 3

For others trying to download files from AWS S3 looking for a more user-friendly solution with other industrial-strength features, check out https://github.com/d6t/d6tpipe. It abstracts the S3 functions into a simpler interface. It also supports directory sync, uploading files, permissions and many other things you need to sync files from S3 (and ftp).

import d6tpipe
api = d6tpipe.api.APILocal() # keep permissions locally for security

settings = \
{
    'name': 'my-files',
    'protocol': 's3',
    'location': 'bucket-name',
    'readCredentials' : {
        'aws_access_key_id': 'AAA',
        'aws_secret_access_key': 'BBB'
    }
}

d6tpipe.api.create_pipe_with_remote(api, settings)

pipe = d6tpipe.Pipe(api, 'my-files')
pipe.scan_remote() # show all files
pipe.pull_preview() # preview
pipe.pull(['k.png']) # download single file
pipe.pull() # download all files

pipe.files() # show files
file=open(pipe.dirpath/'k.png') # access file

Solution 4

You can setup your AWS profile with awscli to avoid introduce your credentials in the file. First add your profile:

aws configure --profile account1

Then in your code add:

aws_session = boto3.Session(profile_name="account1")
s3_client = aws_session.client('s3')

View more solutions

Comments

[PolarBear10]

@Joaquin please add documentation to where you found this for a more complete answer. Nevertheless +1

[Robert King]

For your use of download_fileobj may I suggest you add the line: f.seek(0). The file is left in an non-deterministic state. This line ensures you start reading it back from the beginning (just spent some time figuring this out myself!)

[ramhiser]

@RobertKing do you mind elaborating on this point? Why add f.seek(0)?