How to check if specific resource already exists in CloudFormation script

Question

I am using cloudformation to create a stack which inlcudes an autoscaled ec2 instance and an S3 bucket. For the S3 bucket I have DeletionPolicy set to Retain, which works fine, until I want to rerun my cloudformation script again. Since on previous runs, the script created the S3 bucket, it fails on subsequent runs saying my S3 bucket already exists. None of the other resources of course get created as well. My question is how do I check if my S3 bucket exists first inside the cloudformation script, and if it does, then skip creating that resources. I've looked in conditions in the AWS, but it seems all parameter based, I have yet to find a function which checks from existing resources.

Solution 1

There is no obvious way to do this, unless you create the template dynamically with an explicit check. Stacks created from the same template are independent entities, and if you create a stack that contains a bucket, delete the stack while retaining the bucket, and then create a new stack (even one with the same name), there is no connection between this new stack and the bucket created as part of the previous stack.

If you want to use the same S3 bucket for multiple stacks (even if only one of them exists at a time), that bucket does not really belong in the stack - it would make more sense to create the bucket in a separate stack, using a separate template (putting the bucket URL in the "Outputs" section), and then referencing it from your original stack using a parameter.

Update November 2019:

There is a possible alternative now. On Nov 13th AWS launched CloudFormation Resource Import. With that feature you can now creating a stack from existing resources. Currently not many resource types are supported by this feature, but S3 buckets are.

In your case you'd have to do it in two steps:

1. Create a template that only contains the preexisting S3 bucket using the "Create stack" > "With existing resources (import resources)" (this is the --change-set-type IMPORT flag in the CLI) (see docs)
2. Update the the template to include all resources that don't already exist.

As they note in their documentation; this feature is very versatile. So it opens up a lot of possibilities. See docs for more info.

Solution 2

Using cloudformation you can use Conditions I created an input parameter "ShouldCreateBucketInputParameter" and then using CLI you just need to set "true" or "false"

Cloudformation json file:

{
"AWSTemplateFormatVersion": "2010-09-09",
"Transform": "AWS::Serverless-2016-10-31",
"Description": "",
"Parameters": {
    "ShouldCreateBucketInputParameter": {
      "Type": "String",
      "AllowedValues": [
        "true",
        "false"
      ],
      "Description": "If true then the S3 bucket that will be proxied will be created with the CloudFormation stack."
    }
},
"Conditions": {
  "CreateS3Bucket": {
    "Fn::Equals": [
      {
        "Ref": "ShouldCreateBucketInputParameter"
      },
      "true"
    ]
  }
},
"Resources": {
    "SerialNumberBucketResource": {
        "Type": "AWS::S3::Bucket",
        "Condition": "CreateS3Bucket",
        "Properties": {
            "AccessControl": "Private"
        }
    }
},
"Outputs": {}
}

And then (I am using CLI do deploy the stack)

aws cloudformation deploy --template ./s3BucketWithCondition.json --stack-name bucket-stack --parameter-overrides ShouldCreateBucketInputParameter="true" S3BucketNameInputParameter="BucketName22211"

Solution 3

Just add an input parameter to the CloudFormation template to indicate that an existing bucket should be used.... unless you don't already know at the time when you are going to use the template? Then you can either add a new resource or not based on the parameter value.

Comments

[MikeW]

Yes, this is not such a problem in Terraform.

[alexfvolk]

The goal is to create a stack for a Docker registry - the bucket to serve as backend to store the images, and the ec2 host to serve as the docker registry itself, plus all the necessary IAM policies and security groups. In my mind, it is one entity, thus I was hoping to create 1 script that is flexible enough. I think I will just create a condition and input a parameter using Amazon CLI scripts when I run it, that is the only way I see how you could make the cloudfomration script fully automated and re-runnable.

[Dreampuf]

It seems like Cloudformation aims at an API set to make the AWS resource initialization easily rather than a states management tool.

[Master James]

Maybe try Hashicorp's Terraform. At least that's what I would probably try today.

[Vincent Buscarello]

What would this parameter be? Sorry pretty new :)

[DeadlyChambers]

Yeah, I gotta go with @VincentBuscarello why would you not at least throw out some code. Didn't know my project manager was throwing out advice on SO

[DeadlyChambers]

Well done, greatly appreciate your contributions Jorge.

[hyprstack]

This should be the accepted answer