Error deleting Target Group: ResourceInUse when changing target ports in AWS through Terraform

Question

I am currently working through the beta book "Terraform Up & Running, 2nd Edition". In chapter 2, I created an auto scaling group and a load balancer in AWS.

Now I made my backend server HTTP ports configurable. By default they listen on port 8080.

variable "server_port" {
    …
    default = 8080
}

resource "aws_launch_configuration" "example" {
    …
    user_data = <<-EOF
                #!/bin/bash
                echo "Hello, World" > index.html
                nohup busybox httpd -f -p ${var.server_port} &
                EOF
    …
}

resource "aws_security_group" "instance" {
    …
    ingress {
        from_port = var.server_port
        to_port = var.server_port
        …
    }
}

The same port also needs to be configured in the application load balancer's target group.

resource "aws_lb_target_group" "asg" {
    …
    port = var.server_port
    …
}

When my infrastructure is already deployed, for example with the configuration for the port set to 8080, and then I change the variable to 80 by running terraform apply --var server_port=80, the following error is reported:

> Error: Error deleting Target Group: ResourceInUse: Target group
> 'arn:aws:elasticloadbalancing:eu-central-1:…:targetgroup/terraform-asg-example/…'
> is currently in use by a listener or a rule   status code: 400,

How can I refine my Terraform infrastructure definition to make this change possible? I suppose it might be related to a lifecycle option somewhere, but I didn't manage to figure it out yet.

---

For your reference I attach my whole infrastructure definition below:

provider "aws" {
    region = "eu-central-1"
}

output "alb_location" {
    value = "http://${aws_lb.example.dns_name}"
    description = "The location of the load balancer"
}

variable "server_port" {
    description = "The port the server will use for HTTP requests"
    type = number
    default = 8080
}

resource "aws_lb_listener_rule" "asg" {
    listener_arn = aws_lb_listener.http.arn
    priority = 100

    condition {
        field = "path-pattern"
        values = ["*"]
    }

    action {
        type = "forward"
        target_group_arn = aws_lb_target_group.asg.arn
    }
}

resource "aws_lb_target_group" "asg" {
    name = "terraform-asg-example"
    port = var.server_port
    protocol = "HTTP"
    vpc_id = data.aws_vpc.default.id

    health_check {
        path = "/"
        protocol = "HTTP"
        matcher = "200"
        interval = 15
        timeout = 3
        healthy_threshold = 2
        unhealthy_threshold = 2
    }
}

resource "aws_lb_listener" "http" {
    load_balancer_arn = aws_lb.example.arn
    port = 80
    protocol = "HTTP"

    default_action {
        type = "fixed-response"

        fixed_response {
            content_type = "text/plain"
            message_body = "404: page not found"
            status_code = 404
        }
    }
}

resource "aws_lb" "example" {
    name = "terraform-asg-example"
    load_balancer_type = "application"
    subnets = data.aws_subnet_ids.default.ids
    security_groups = [aws_security_group.alb.id]
}

resource "aws_security_group" "alb" {
    name = "terraform-example-alb"

    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

resource "aws_autoscaling_group" "example" {
    launch_configuration = aws_launch_configuration.example.name
    vpc_zone_identifier = data.aws_subnet_ids.default.ids

    target_group_arns = [aws_lb_target_group.asg.arn]
    health_check_type = "ELB"

    min_size = 2
    max_size = 10

    tag {
        key = "Name"
        value = "terraform-asg-example"
        propagate_at_launch = true
    }
}

resource "aws_launch_configuration" "example" {
    image_id = "ami-0085d4f8878cddc81"
    instance_type = "t2.micro"
    security_groups = [aws_security_group.instance.id]

    user_data = <<-EOF
                #!/bin/bash
                echo "Hello, World" > index.html
                nohup busybox httpd -f -p ${var.server_port} &
                EOF
    lifecycle {
        create_before_destroy = true
    }
}

resource "aws_security_group" "instance" {
    name = "terraform-example-instance"

    ingress {
        from_port = var.server_port
        to_port = var.server_port
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

data "aws_subnet_ids" "default" {
    vpc_id = data.aws_vpc.default.id
}

data "aws_vpc" "default" {
    default = true
}

Solution 1

From the issue link in the comment on Cannot rename ALB Target Group if Listener present:

Add a lifecycle rule to your target group so it becomes:

resource "aws_lb_target_group" "asg" {
    name     = "terraform-asg-example"
    port     = var.server_port
    protocol = "HTTP"
    vpc_id   = data.aws_vpc.default.id

    health_check {
      path                = "/"
      protocol            = "HTTP"
      matcher             = "200"
      interval            = 15
      timeout             = 3
      healthy_threshold   = 2
      unhealthy_threshold = 2
    }

    lifecycle {
      create_before_destroy = true
    }
}

However you will need to choose a method for changing the name of your target group as well. There is further discussion and suggestions on how to do this.

But one possible solution is to simply use a guid but ignore changes to the name:

resource "aws_lb_target_group" "asg" {
    name     = "terraform-asg-example-${substr(uuid(), 0, 3)}"
    port     = var.server_port
    protocol = "HTTP"
    vpc_id   = data.aws_vpc.default.id

    health_check {
      path                = "/"
      protocol            = "HTTP"
      matcher             = "200"
      interval            = 15
      timeout             = 3
      healthy_threshold   = 2
      unhealthy_threshold = 2
    }

    lifecycle {
      create_before_destroy = true
      ignore_changes        = [name]
    }
}

Solution 2

Slightly simpler than @FGreg's solution, add a lifecycle policy and switch from name to name_prefix which will prevent naming collisions.

resource "aws_lb_target_group" "asg" {
    name_prefix = "terraform-asg-example"
    port = var.server_port
    protocol = "HTTP"
    vpc_id = data.aws_vpc.default.id
    lifecycle {
        create_before_destroy = true
    }

    health_check {
        path = "/"
        protocol = "HTTP"
        matcher = "200"
        interval = 15
        timeout = 3
        healthy_threshold = 2
        unhealthy_threshold = 2
    }
}

No need for uuid or ignore_changes settings.

Comments

[alonisser]

Thanks for the lifecycle and random name tip!

[UnsafePointer]

This is not a good solution, because whenever you run terraform apply again, the expression ${substr(uuid(),0, 3)} will be evaluated again and it will give you a different value, forcing a create/destroy cycle again. A better solution is to rename the resource manually.

[FGreg]

@Ruenzuo that is what the lifecycle{ ignore_changes = [name] } does. It tells terraform to ignore changes to the name of the resource when calculating if the resource should be replaced or not. I agree, this isn't a great solution but it's the best working solution I've been able to find

[jchook]

name_prefix currently has a 6 character limit due to the long suffix.

[CryptoFool]

This fixed the issue for me. Thanks! This would be a better solution if not for the severe limitation on the max prefix length. Although....I guess it doesn't really matter, as each name is different anyway. I suppose it really just matters when you see the name when perusing AWS resources. This might be a place where a tag would be useful to provide an alternate name.