How to for_each through a list(objects) in Terraform 0.12

Question

I need to deploy a list of GCP compute instances. How do I loop for_each through the "vms" in a list of objects like this:

    "gcp_zone": "us-central1-a",
    "image_name": "centos-cloud/centos-7",
    "vms": [
      {
        "hostname": "test1-srfe",
        "cpu": 1,
        "ram": 4,
        "hdd": 15,
        "log_drive": 300,
        "template": "Template-New",
        "service_types": [
          "sql",
          "db01",
          "db02"
        ]
      },
      {
        "hostname": "test1-second",
        "cpu": 1,
        "ram": 4,
        "hdd": 15,
        "template": "APPs-Template",
        "service_types": [
          "configs"
        ]
      }
    ]    
}

Solution 1

Seem's like I found what to do. If you pass not the maps of maps but the list of maps you can use such code

resource "google_compute_instance" "node" {
    for_each = {for vm in var.vms:  vm.hostname => vm}

    name         = "${each.value.hostname}"
    machine_type = "custom-${each.value.cpu}-${each.value.ram*1024}"
    zone         = "${var.gcp_zone}"

    boot_disk {
        initialize_params {
        image = "${var.image_name}"
        size = "${each.value.hdd}"
        }
    }

    network_interface {
        network = "${var.network}"
    }

    metadata = {
        env_id = "${var.env_id}"
        service_types = "${join(",",each.value.service_types)}"
  }
}

It will create actual number of instance and when you remove for example middle one of three(if you create three:)), terraform will remove what we asked.

Solution 2

I work a lot with iterators in Terraform, they always gave me bad headaches. Therefore I identified three of the most common iterator patterns (code examples are given below), which helped me construct a lot of nice modules (source).

• Using for_each on a list of strings
• Using for_each on a list of objects
• Using for_each as a conditional

Using for_each and a list of strings is the easiest to understand, you can always use the toset() function. When working with a list of objects you need to convert it to a map where the key is a unique value. The alternative is to put a map inside your Terraform configuration. Personally, I think it looks cleaner to have a list of objects instead of a map in your configuration. The key usually doesn't have a purpose other than to identify unique items in a map, which can thus be constructed dynamically. I also use iterators to conditionally deploy a resource or resource block, especially when constructing more complex modules.

Using for_each on a list of strings

locals {
  ip_addresses = ["10.0.0.1", "10.0.0.2"]
}

resource "example" "example" {
  for_each   = toset(local.ip_addresses)
  ip_address = each.key
}

Using for_each on a list of objects

locals {
  virtual_machines = [
    {
      ip_address = "10.0.0.1"
      name       = "vm-1"
    },
    {
      ip_address = "10.0.0.1"
      name       = "vm-2"
    }
  ]
}    

resource "example" "example" {
  for_each   = {
    for index, vm in local.virtual_machines:
    vm.name => vm # Perfect, since VM names also need to be unique
    # OR: index => vm (unique but not perfect, since index will change frequently)
    # OR: uuid() => vm (do NOT do this! gets recreated everytime)
  }
  name       = each.value.name
  ip_address = each.value.ip_address
}

Using for_each as a conditional

variable "deploy_something" {
  type        = bool
  description = "Indicates whether to deploy something."
  default     = true
}

# Using count and a conditional, for_each is also possible here.
# See the next solution using a for_each with a conditional.
resource "example" "example" {
  count      = var.deploy_example ? 0 : 1
  name       = ...
  ip_address = ...
}

variable "enable_something" {
  type        = bool
  description = "Indicates whether to enable something."
  default     = false
}

resource "example" "example" {
  name       = ...
  ip_address = ...

  # Note: dynamic blocks cannot use count!
  # Using for_each with an empty list and list(1) as a readable alternative. 
  dynamic "logs" {
    for_each = var.enable_logs ? [] : [1]
    content {
      name     = "logging"
    }
  }
}

Solution 3

From Terraform 0.12, you can use the for_each with modules like the following:

modules/google_compute_instance/variables.tf

variable "hosts" {
    type = map(object({
        hostname        = string
        cpu             = number
        ram             = number
        hdd             = number
        log_drive       = number
        template        = string 
        service_types   = list(string)
      }))
    }

modules/google_compute_instance/main.tf

resource "google_compute_instance" "gcp_instance" {
  for_each = var.hosts

  hostname      = each.value.repository_name
  cpu           = each.value.cpu
  ram           = each.value.ram
  hdd           = each.value.hdd
  log_drive     = each.value.log_drive
  template      = each.value.template
  service_types = each.value.service_types
}

#servers.tf

module "gcp_instances" {
    source = "./modules/google_compute_instance"

    hosts = {
        "test1-srfe" = {
            hostname        = "test1-srfe",
            cpu             = 1,
            ram             = 4,
            hdd             = 15,
            log_drive       = 300,
            template        = "Template-New",
            service_types   = ["sql", "db01", "db02"]
        },
        "test1-second" = {
            hostname        = "test1-second",
            cpu             = 1,
            ram             = 4,
            hdd             = 15,
            log_drive       = 300,
            template        = "APPs-Template",
            service_types   = ["configs"]
        },
    }
}

Of course, you can add as many variables as needed and use them in the module.

Solution 4

You can do the following:

for_each = toset(keys({for i, r in var.vms:  i => r}))
cpu = var.vms[each.value]["cpu"]

Assuming you had the following:

variable "vms" {
    type = list(object({
        hostname        = string
        cpu             = number
        ram             = number
        hdd             = number
        log_drive       = number
        template        = string 
        service_types   = list(string)
    }))
    default = [
        {
            cpu: 1
            ...
        }
    ]
}

Solution 5

Using the for_each block is pretty new and there's not too much documentation. Some of the best info comes from their announcement blog post: https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/

Also make sure to check out the Dynamic Blocks section of their documentation: https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

From what your example looks like you need to have a set of values for each instance that is created so you'll have a map of maps:

Below is an example I created using Terraform 0.12.12:

variable "hostnames" {
    default = {
        "one" = {
            "name" = "one",
            "machine" = "n1-standard-1",
            "os" = "projects/coreos-cloud/global/images/coreos-stable-2247-5-0-v20191016",
            "zone" = "us-central1-a"
        },
        "two" = {
            "name" = "two",
            "machine" = "n1-standard-2",
            "os" = "projects/centos-cloud/global/images/centos-8-v20191018",
            "zone" = "us-central1-b"
        }
    }
}

resource "google_compute_instance" "default" {
    for_each = var.hostnames
    name         = each.value.name
    machine_type = each.value.machine
    zone         = each.value.zone

    boot_disk {
        initialize_params {
            image = each.value.os
        }
    }

    scratch_disk {
    }

    network_interface {
        network = "default"
    }
}

Terraform plan output:

Terraform will perform the following actions:

  # google_compute_instance.default["one"] will be created
  + resource "google_compute_instance" "default" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "n1-standard-1"
      + metadata_fingerprint = (known after apply)
      + name                 = "one"
      + project              = (known after apply)
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + zone                 = "us-central1-a"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image  = "projects/coreos-cloud/global/images/coreos-stable-2247-5-0-v20191016"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = "default"
          + network_ip         = (known after apply)
          + subnetwork         = (known after apply)
          + subnetwork_project = (known after apply)
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + scratch_disk {
          + interface = "SCSI"
        }
    }

  # google_compute_instance.default["two"] will be created
  + resource "google_compute_instance" "default" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "n1-standard-2"
      + metadata_fingerprint = (known after apply)
      + name                 = "two"
      + project              = (known after apply)
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + zone                 = "us-central1-b"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image  = "projects/centos-cloud/global/images/centos-8-v20191018"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = "default"
          + network_ip         = (known after apply)
          + subnetwork         = (known after apply)
          + subnetwork_project = (known after apply)
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + scratch_disk {
          + interface = "SCSI"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

View more solutions

Comments

[TheShadow2707]

So I need to convert list to map, as I undersand it change the index of list to key and it's gonna be map.

[TheShadow2707]

It's nice but variable is dynamically provided each time.

[Tamás Juhász]

Common variables can be added separately as variables and reused as var.variable. The unique ones have to be declared separately anyway.

[svinther]

This works well. The list(object)) is converted to a map that can be used as the for_each value. Just select a key (e.g hostname) that is unique

[pijemcolu]

You're a hero. Finally I've found a proper example, from start to end.

[user3508953]

Care to share what : vm.hostname => vm does in the for_each line: for_each = {for vm in var.vms: vm.hostname => vm}?

[Ian Walker-Sperber]

@user3508953 - This is a way of setting the "key" and "value" for the resource, where vm.hostname is the key and vm is the value. So in this example, we uniquely identify each resource in the loop by hostname. If you had multiple VMs in our list with the same hostname but different CPUs, then we might rewrite as "${vm.hostname}:${vm.cpu}" => vm. This allows terraform to track the resources by a key you define... IIRC Terraform tracks by index otherwise (which can be problematic if you reorder things)