How can I reuse existing resources in CloudFormation?

Question

I have an S3 bucket as a resource in my CloudFormation template, with DeletionPolicy set to Retain. This works as expected: when deleting the stack, it does indeed retain the bucket. However, when I attempt to create the stack again, creation fails while attempting to create the same bucket again, with an error message complaining that it already exists.

What do I need to add to my CloudFormation template to make it not try recreating a resource which already exist?

Relevant fragment of my template is as follows:

      "Resources": {
        "SomeS3Bucket" : {
          "Type" : "AWS::S3::Bucket",
          "DeletionPolicy" : "Retain",
          "Properties": {
              "BucketName": "SomeS3Bucket"
              }
          }

Solution 1

One approach is to add an input parameter to the CloudFormation template to indicate that an existing bucket should be used.

Use Condition clauses in the template to create the bucket only if the parameter indicates it is needed.

Solution 2

CloudFormation uses tags with the "aws:" prefix to keep track of what resources are associated with what entries in which stacks -- that's the "live" state it uses to compare with a template before deciding what to add/delete/update.

As a user, you can't add, edit, or delete such tags.

So if your existing resources don't have these tags, or don't have the correct values for those tags, then they aren't considered part of the new stack, and I don't see a way to change that.

Solution 3

I am trying to automate this too, as it seems it cannot be done just with Cloudformation template. The process I am thinking of would:

1. create another temporary bucket temp-$originalbucketname
2. copy all the content there bucket-to-bucket to save time
3. remove all the content from $originalbucketname
4. remove $originalbucketname now that it is empty
5. create the Cloudformation stack (which will recreate the bucket)
6. copy the content back
7. remove temp-$originalbucketname

That's a very involved process, depending on the bucket size it could easily take hours as most of the steps are O(n) with the number of keys.

You would think Cloudformation is the basic layer of AWS automation, but I think it's just a (pretty limited) monster pulling together byzantine APIs for all their services.

Comments

[Drew Khoury]

out of interest, what's the bucket used for? Maybe there's another way around it depending on what you're trying to do.

[Tim Ludwinski]

AWS apparently added the ability to import existing resources recently. Not quite sure how it works or if it will be useful in this case but here's the link: docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/…

[Misha Narinsky]

+1 as so far this is the only way I've seen. Not marking it as answer though, because I'm really looking at a way to automate that.

[jgomo3]

There should be a way, if not: How does "CloudFormer" works?