boto issue with IAM role

Question

I'm trying to use AWS' recently announced "IAM roles for EC2" feature, which lets security credentials automatically get delivered to EC2 instances. (see http://aws.amazon.com/about-aws/whats-new/2012/06/11/Announcing-IAM-Roles-for-EC2-instances/).

I've set up an instance with an IAM role as described. I can also get (seemingly) proper access key / credentials with curl.

However, boto fails to do a simple call like "get_all_buckets", even though I've turned on ALL S3 permissions for the role.

The error I get is "The AWS Access Key Id you provided does not exist in our records"

However, the access key listed in the error matches the one I get from curl.

Here is the failing script, run on an EC2 instance with an IAM role attached that gives all S3 permissions:

import urllib2
import ast
from boto.s3.connection import S3Connection

resp=urllib2.urlopen('http://169.254.169.254/latest/meta-data/iam/security-credentials/DatabaseApp').read()
resp=ast.literal_eval(resp)
print "access:" + resp['AccessKeyId']
print "secret:" + resp['SecretAccessKey']
conn = S3Connection(resp['AccessKeyId'], resp['SecretAccessKey'])
rs= conn.get_all_buckets()

Solution 1

If you are using boto 2.5.1 or later it's actually much easier than this. Boto will automatically find the credentials in the instance metadata for you and use them as long as no other credentials are found in environment variables or in a boto config file. So, you should be able to simply do this on the EC2 instance:

>>> import boto
>>> c = boto.connect_s3()
>>> rs = c.get_all_buckets()

The reason that your manual approach is failing is that the credentials associated with the IAM Role is a temporary session credential and consists of an access_key, a secret_key and a security_token and you need to supply all three of those values to the S3Connection constructor.

Solution 2

I don't know if this answer will help anyone but I was getting the same error, I had to solve my problem a little differently. First, my amazon instance did not have any IAM roles. I thought I could just use the access key and the secret key but I kept getting this error with only those two keys. I read I needed a security token as well, but I didn't have one because I didn't have any IAM roles. This is what I did to correct the issue:

1. Create an IAM role with AmazonS3FullAccess permissions.
2. Start a new instance and attach my newly created role.

3. Even after doing this it still didn't work. I had to also connect to the proper region with the code below:

   import boto.s3.connection
   conn = boto.s3.connect_to_region('your-region')
   conn.get_all_buckets()