Boto3 error: The AWS Access Key Id you provided does not exist in our records

Question

I am currently trying to get access to Amazon S3 inside a virtual machine and download files like so:

s3 = boto3.resource('s3',
         aws_access_key_id="xxxxxxxxxxx",
         aws_secret_access_key="xxxxxxxxxxxxxxxxx")
s3client = boto3.client('s3')

bucket = s3.Bucket('bucketone')

for obj in bucket.objects.all():
    s3client.download_file(bucket_name, obj.key, filename)

But I’m getting the error:

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

What could I be doing wrong? I checked my aws_access_key_id and aws_secret_access_key multiple times, but still getting the same error. The same code locally, but not on a virtual machine, actually works on a different computer as well. There is a reason why I’m hardcoding in the keys, as I have to.

Solution 1

You need to set the access for the boto3 session. You don't really want to put your keys in your code. What I would recommend doing first is running 'aws configure' and setting your aws_access_key_id and aws_secret_access_key in your .credentials file. Then in your code do the following:

session = boto3.Session(profile_name='name_of_your_profile')

If you have just the default profile, you might not need to do that or for good measure, just put:

session = boto3.Session(profile_name='default')

Once you have that in your code you can establish a connection to s3 with:

s3 = session.resource('s3')
bucket = s3.Bucket('bucketone')
for obj in bucket.objects.all():
   print(obj.key)

There is some problem with your code as well. You are creating an s3 client. S3 client does not have a Bucket method or property. To do the same thing with the s3 client you would do:

s3client = session.client('s3')
response = s3client.get_object(Bucket='bucketone', key='your key')

You can then iterate through the response that is returned to see the list of objects in the bucket.

That should take care of your error.

Solution 2

Boto3 users BEWARE

TL;DR

If you are using temporary credentials to connect to AWS services through Boto3, you MUST include a current aws_session_token as a parameter to your boto3.session.Session instance.

from boto3.session import Session

# Ideally this is picked up your ENV.
id_ = "<id>"
secret = "<secret>"
token = "token"

session = Session(
                aws_access_key_id=id_,
                aws_secret_access_key=secret,
                aws_session_token=token,
                region_name='<region>'
            )

# Test it on a service (yours may be different)
s3 = session.resource('s3')

# Print out bucket names
for bucket in s3.buckets.all():
    print(bucket.name)

Explanation

This is a crucial piece of information when you are testing credentials in Boto3: The error you receive may say this,

ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

but may mean you are missing an aws_session_token if you are using temporary credentials (in my case, role-based credentials).

According to AWS documentation, these are the parameters available to a boto3.session.Session object, however, there is no indication or clarification when it comes to this behavior in Boto3:

Parameters
aws_access_key_id (string) -- AWS access key ID
aws_secret_access_key (string) -- AWS secret access key
aws_session_token (string) -- AWS temporary session token
region_name (string) -- Default region when creating new connections
botocore_session (botocore.session.Session) -- Use this Botocore session instead of creating a new default one.
profile_name (string) -- The name of a profile to use. If not given, then the default profile is used.

Regarding the aws_session_token

Specifies an AWS session token used as part of the credentials to authenticate the user. A session token is required only if you manually specify temporary security credentials.

Resources

• aws_session_token
• Common scenarios for roles: Users, applications, and services
• Boto3 Credentials
• Session Reference

Solution 3

Initial answer: If you're using EC2/Lambda/etc. you'll get temporary security credentials which means you'll need to use AWS_SESSION_TOKEN environment variable and pass it to boto3.

Omitting it will throw InvalidAccessKeyId error.

Update: I can't remember exactly, but I think thr cause of my problem back then was not due to AWS_SESSION_TOKEN at all but because I bundled my own boto3 lib to Lambda app (using Zappa). After I "removed" boto3 (hence using Lambda's built-in boto3) the problem was gone.

Solution 4

You need to set up the AWS Command-line interface and configure its settings. 1. Download and install AWS CLI. 2. Run aws configure` command and provide access key and secret key. If you still get an error then check whether the IAM user's programmatic access is active or not. Delete the existing access key and create a new one for use.

View more solutions