Getting files from an s3 bucket using IAM role credentials

Question

I am trying to retrieve some files from a private s3 bucket to a filesystem location elastic beanstalk ec2 instance, but with no success.

I've created a bucket named dev-config containing a file named local.properties.

I've created a IAM policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::dev-config",
                "arn:aws:s3:::dev-config/*"
            ]
        }
    ]
}

And associated that policy to a IAM role, that in turn is associated with the EC2 instance. I have confirmed that I can fetch files from the s3 bucket using the aws-cli without providing any additional credentials. i.e. aws s3 ls s3://dev-config/local.properties

To my project I've added the following file:

.ebextensions/01_files.config

"/usr/share/tomcat7/lib/local.properties" :
    mode: "000777"
    owner: ec2-user
    group: ec2-user
    source: http://s3.amazonaws.com/dev-config/local.properties

I've also tried a few variations of the source url

    source: http://dev-config.s3.amazonaws.com/dev-config/local.properties
    source: http://dev-config.s3.amazonaws.com/local.properties
    source: s3://dev-config/local.properties

And I've also tried adding an authentication attribute with no success (there seem to be no docs on possible values for authentication). authentication: S3Access

None of the approaches have worked so far.

In some cases I get access denied messages in the logs:

    <?xml version="1.0" encoding="UTF-8"?>
      <Error><Code>AccessDenied</Code><Message>Access Denied</Message>
      <RequestId>blahblah</RequestId>
      <HostId>blahblah</HostId>
    </Error>

In other cases I have had error messages in the local.properties file itself PermanentRedirect The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. dev-config dev-config.s3.amazonaws.com
blahlblah blahlblah

Has managed to get this working?

Solution 1

After taking a look at this answer to Using environment properties with files in elastic beanstalk config files I added the following section to the .ebextensions/01_files.config

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Access:
          type: S3
          roleName: aws-elasticbeanstalk-ec2-role
          buckets: dev-config

and updated the s3 url to include the bucket name in the host, so the final file looked like this:

"/usr/share/tomcat7/lib/local.properties" :
    mode: "000777"
    owner: ec2-user
    group: ec2-user
    source: https://dev-config.s3.amazonaws.com/local.properties

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Access:
          type: S3
          roleName: aws-elasticbeanstalk-ec2-role
          buckets: dev-config

This enabled the elastic beanstalk ec2 instance to use the IAM role associated with it to access the s3 bucket containing the files.

PS: For this configuration to work, make sure that you've granted access to the S3 bucket in question to the aws-elasticbeanstalk-ec2-role principal. You can get the ARN from IAM console.

Solution 2

Try with this IAM. It works for me.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::dev-config",
                "arn:aws:s3:::dev-config/*"
            ]
        }
    ]
}

If you need to have read/write/delete permissions you need something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                 "arn:aws:s3:::dev-config",
                 "arn:aws:s3:::dev-config/*"
            ]
        }
    ]
}

Regards.

Comments

[ceejayoz]

s3:* should already cover all of those.

[Strelok]

This just doesn't work for me. Always a 403. If I assign the permission to the role manually it works. Am I missing something else in the file?

[diffa]

Sorry to hear that. I had this working, I'll go back and check the details of the answer against the implementation.

[Strelok]

I think I soved. You had to give aws-elasticbeanstalk-ec2-role access to the s3 bucket! Doh! After you do that this ebextensions config works as this only instructs the deployer to auth with that role.

[diffa]

In original question I mentioned that the once I had created the IAM policy, I then "associated that policy to a IAM role, that in turn is associated with the EC2 instance". The answer implies that is aws-elasticbeanstalk-ec2-role, but I didn't specify it. In practice, I would create another role for your app on elastic beanstalk otherwise all of your EB instances would be able to access the bucket, which might not be what you want.