How to add lifecycle rule to an existing AWS S3 bucket with Terraform

Question

I have an existing S3 bucket and I wish to add "folders" and lifecycle_rules to it.

(I say "folders" because that is how they are represented at the client end, as they are accessed via a Storage Gateway.)

I can create the folders, for example on to hold quarterly backups, like:

resource "aws_s3_bucket_object" "quarterly" {
    bucket  = "${var.bucket_id}"
    acl     = "private"
    key     = "quarterly"
    source  = "/dev/null"
}

But if I try and then add a lifecycle rule, as follows

resource "aws_s3_bucket" "quarterly" {
    bucket  = "${var.bucket_id}"
    acl     = "private"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true
        tags {
            "rule"  = "quarterly"
        }

        expiration {
            days = 92
        }
    }
}

I get an error when I do terraform apply.

* aws_s3_bucket.quarterly: Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
    status code: 409, request id: 702396A7D2FA28BA, host id: IJDA+vszRBYl4zmvW56dSnC2va2qpQXlfgeEL7X1QQHHv8eEaYKvSUCL0ZIj/VsdvQ2hkBLGjAY=

I want to create the bucket first and then add folders and lifecycle rules afterwards, rather than embed the lifecycle rules on creation.

Am I missing something, have I got it wrong?

Thanks for your help!

Solution 1

Creating the bucket first and incrementally updating the configuration should work fine, the net result is that if you were to delete the bucket by other means Terraform would recreate it with all of the rules in place.

It looks like you've lost your terraform.tfstate file so Terraform doesn't know that it has already created your bucket, or you created the bucket outside of Terraform in the first place, so it's trying to create it and failing. Terraform needs to "own" the bucket to be able to update its configuration, i.e. the lifecycle rules.

You should be able to import the existing bucket into your state file with something like

terraform import aws_s3_bucket.quarterly <your bucket ID>

See the bottom of https://www.terraform.io/docs/providers/aws/r/s3_bucket.html

Running Terraform should then show it just updating the lifecycle rules.

Solution 2

As per @bodgit, the method is to embed the lifecycle rules within the "aws_s3_bucket" resource and re-running "terraform apply".

lifecycle_rule clauses can be added to (or removed from) the resource and they are applied to the bucket.

I was looking to separate the rules from the bucket creation, so they could be actioned distinctly, but this'll do.

So, I define the bucket with:

resource "aws_s3_bucket" "bucket" {
    bucket      = "${replace(var.tags["Name"],"/_/","-")}"
    region      = "${var.aws_region}"

    tags        = "${merge(var.tags, map("Name", "${replace(var.tags["Name"],"/_/","-")}"))}"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true

        expiration {
            days = 92
        }
    }

}

When I run "terraform apply" the bucket is created with 1 rule.

I then edit my .tf file and add a 2nd lifecycle_rule.

resource "aws_s3_bucket" "bucket" {
    bucket      = "${replace(var.tags["Name"],"/_/","-")}"
    region      = "${var.aws_region}"

    tags        = "${merge(var.tags, map("Name", "${replace(var.tags["Name"],"/_/","-")}"))}"

    lifecycle_rule {
        id      = "quarterly_retention"
        prefix  = "quarterly/"
        enabled = true

        expiration {
            days = 92
        }
    }

    lifecycle_rule {
        id      = "permanent_retention"
        enabled = true
        prefix  = "permanent/"

        transition {
            days            = 1
            storage_class   = "GLACIER"
        }
    }

}

When I execute "terraform apply" again, the bucket has the 2nd rule added.

If I then edit the rule again, remove the rules, and run an "apply" again, the rules are gone.

If I then edit, add a rule back, and run apply again, that rule is there.

Thanks for the help!

Comments

[prowla]

Thanks for that. I am creating the bucket in Terraform. If I embed the lifecycle rules into the aws_s3_bucket resource creation it works fine and creates the bucket with the rules.

[bodgit]

You can't have two aws_s3_bucket resources referencing the same S3 bucket, but you can create the bucket and later amend the same resource with lifecycle rules and they'll be added to the existing bucket.