Amazon Cloud Formation: Import file from S3 bucket

amazon-ec2 amazon-web-services amazon-s3 amazon-cloudformation

14,669

Solution 1

I don't think you are getting any files copied over. You might need to add the IAM S3 access role to your instance - for example AmazonS3FullAccess. I do something similar but I set up my instances with that role and copy files to it from S3 with aws s3 cp ... or entire directories with aws s3 sync .... I put these commands in the User data field so I can "set up the instance and forget about it"

Solution 2

To get S3 file copy working with S3 Readonly access you need:

To assign your instance to an Instance Profile - attached to an Instance Role, with read only access to the bucket - [ "s3:Get*", "s3:List*" ]
Define AWS::CloudFormation::Authentication next to your AWS::CloudFormation::Init section and configure the role like below.

Ensure your "source" section looks like below.

"AWS::CloudFormation::Authentication" : {
  "default" : {
    "type" : "S3",
    "buckets" : [ { "Ref" : "ConfigBucket" } ],
    "roleName" : { "Ref" : "MyNodeRole" }
    }
},

"AWS::CloudFormation::Init" : {
  "config" : {
    "files" : {
      "/etc/myapp/config/filename": {
        "source" : { "Fn::Join": [ "", [ "http://", { "Ref" : "ConfigBucket" }, ".s3-us-west-1.amazonaws.com/config/filename" ]]},
        "mode": "000444",
        "owner": "root",
        "group": "root",
        "authentication": "default"
        }
    }
  }
}

The source link needs to be a key. AWS Documentation of an S3 bucket states:

Objects stored in the buckets have a unique key value and are retrieved using a HTTP URL address. For example, if an object with a key value /photos/mygarden.jpg is stored in the myawsbucket bucket, then it is addressable using the URL http://myawsbucket.s3.amazonaws.com/photos/mygarden.jpg.

14,669

Soatl

I am a polyglot developer with a Masters in Computer Science from Georgia Tech. I focus on full-stack development and enjoy learning about cybersecurity principles and machine learning. I also have experience with big data as well as DevOps. #SOreadytohelp

Updated on September 18, 2022

Comments

Soatl over 1 year

I am struggling to get files from my S3 bucket to my EC2 instance using cloud formation. I have the following files:

"files" : {
  "/var/www/text.txt": {
      "source": "https://s3.amazonaws.com/bucket/test.txt",
      "mode": "000644",
      "owner" : "root",
      "group" : "root"
  },
  "/var/temp/http.conf": {
    "source" : "https://s3.amazonaws.com/bucket/httpd.conf",
    "mode": "000644",
    "owner" : "root",
    "group" : "root"
  },
  "/var/temp/mime-types.conf": {
    "source" : "https://s3.amazonaws.com/bucket/mime-types.conf",
    "mode": "000644",
    "owner" : "root",
    "group" : "root"
  }
}

Where test.txt is a file with the text "text", and the other two are config files.

I have updated my permissions on the test.txt file so that anyone can view or download it, have tried making it http over https and have tried to remove the other two files.

So far, /var/www/ will be created with four directories, but not /var/temp/. /var/www/error will also contain files.

/var/www/error/

I am under the impression this should just work, but that is not the case. Am I doing something wrong? I am following the aws documentation.

EDIT I looked into the error log. Here is the output. It apparently cannot find the file:

2015-06-04 14:28:50,381 [ERROR] HTTP Error 404 : <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 159, in _retry
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 283, in get_role_creds
    resp.raise_for_status()
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 404 Client Error: Not Found
2015-06-04 14:28:51,208 [ERROR] HTTP Error 404 : <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 159, in _retry
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 283, in get_role_creds
"cfn-init.log" [readonly] 1216L, 58277C

Michael - sqlbot almost 9 years

Once the instance is up, can you access the files in S3 manually, from the instance, with curl or wget? If not, your networking configuration will be the issue, not confirmation. Also, those URLs would only work if your bucket is in the US-Standard region.
Bazze almost 9 years

Could you have a look in the cfn-init log file? You should be able to see what the issue is in there. It's located here: /var/log/cfn-init.log.
Soatl almost 9 years

I made an edit to show the error. @Michael, it would be more ideal for us to not have to manually go into this instance and get the files, we want to pretty much set up this instance and forget about it.
Michael - sqlbot almost 9 years

@PepperedLemons, certainly. I didn't mean to imply that you should consider that as an alternative... I was thinking of it as a diagnostic step only, to confirm S3 connectivity and object accessibility.

Soatl almost 9 years

That worked! I would have rather used the "file" portion of the template, but instead I just added the aws s3 copies to the user data. Thanks!
RalfFriedl over 4 years

Do you have a reason to believe that is the OP's problem?