Initial setup of terraform backend using terraform

Question

I'm just getting started with terraform and I'd like to be able to use AWS S3 as my backend for storing the state of my projects.

terraform {
    backend "s3" {
      bucket = "tfstate"
      key = "app-state"
      region = "us-east-1"
    }
}

I feel like it is sensible to setup my S3 bucket, IAM groups and polices for the backend storage infrastructure with terraform as well.

If I setup my backend state before I apply my initial terraform infrastructure, it reasonably complains that the backend bucket is not yet created. So, my question becomes, how do I setup my terraform backend with terraform, while keeping my state for the backend tracked by terraform. Seems like a nested dolls problem.

I have some thoughts about how to script around this, for example, checking to see if the bucket exists or some state has been set, then bootstrapping terraform and finally copying the terraform tfstate up to s3 from the local file system after the first run. But before going down this laborious path, I thought I'd make sure I wasn't missing something obvious.

Solution 1

To set this up using terraform remote state, I usually have a separate folder called remote-state within my dev and prod terraform folder.

The following main.tf file will set up your remote state for what you posted:

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "tfstate"
     
  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
    bucket = aws_s3_bucket.terraform_state.id

    versioning_configuration {
      status = "Enabled"
    }
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "app-state"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Then get into this folder using cd remote-state, and run terraform init && terraform apply - this should only need to be run once. You might add something to bucket and dynamodb table name to separate your different environments.

Solution 2

Building on the great contribution from Austin Davis, here is a variation that I use which includes a requirement for data encryption:

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "tfstate"

  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "app-state"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = "${aws_s3_bucket.terraform_state.id}"
  policy =<<EOF
{
  "Version": "2012-10-17",
  "Id": "RequireEncryption",
   "Statement": [
    {
      "Sid": "RequireEncryptedTransport",
      "Effect": "Deny",
      "Action": ["s3:*"],
      "Resource": ["arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}/*"],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      },
      "Principal": "*"
    },
    {
      "Sid": "RequireEncryptedStorage",
      "Effect": "Deny",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}/*"],
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      },
      "Principal": "*"
    }
  ]
}
EOF
}

Solution 3

As you've discovered, you can't use terraform to build the components terraform needs in the first place.

While I understand the inclination to have terraform "track everything", it is very difficult, and more headache than it's worth.

I generally handle this situation by creating a simple bootstrap shell script. It creates things like:

1. The s3 bucket for state storage
2. Adds versioning to said bucket
3. a terraform IAM user and group with certain policies I'll need for terraform builds

While you should only need to run this once (technically), I find that when I'm developing a new system, I spin up and tear things down repeatedly. So having those steps in one script makes that a lot simpler.

I generally build the script to be idempotent. This way, you can run it multiple times without concern that you're creating duplicate buckets, users, etc

Solution 4

I created a terraform module with a few bootstrap commands/instructions to solve this:

https://github.com/samstav/terraform-aws-backend

There are detailed instructions in the README, but the gist is:

# conf.tf

module "backend" {
  source         = "github.com/samstav/terraform-aws-backend"
  backend_bucket = "terraform-state-bucket"
}

Then, in your shell (make sure you haven't written your terraform {} block yet):

terraform get -update
terraform init -backend=false
terraform plan -out=backend.plan -target=module.backend
terraform apply backend.plan

Now write your terraform {} block:

# conf.tf

terraform {
  backend "s3" {
    bucket         = "terraform-state-bucket"
    key            = "states/terraform.tfstate"
    dynamodb_table = "terraform-lock"
  }
}

And then you can re-init:

terraform init -reconfigure

Solution 5

Setting up a Terraform backend leveraging an AWS s3 bucket is relatively easy.

First, create a bucket in the region of your choice (eu-west-1 for the example), named terraform-backend-store (remember to choose a unique name.)

To do so, open your terminal and run the following command, assuming that you have properly set up the AWS CLI (otherwise, follow the instructions at the official documentation):

aws s3api create-bucket --bucket terraform-backend-store \
    --region eu-west-1 \
    --create-bucket-configuration \
    LocationConstraint=eu-west-1
# Output:
{
    "Location": "http://terraform-backend-store.s3.amazonaws.com/"
}

The command should be self-explanatory; to learn more check the documentation here.

Once the bucket is in place, it needs a proper configuration for security and reliability. For a bucket that holds the Terraform state, it’s common-sense enabling the server-side encryption. Keeping it simple, try first AES256 method (although I recommend to use KMS and implement a proper key rotation):

aws s3api put-bucket-encryption \
    --bucket terraform-backend-store \
    --server-side-encryption-configuration={\"Rules\":[{\"ApplyServerSideEncryptionByDefault\":{\"SSEAlgorithm\":\"AES256\"}}]}
# Output: expect none when the command is executed successfully

Next, it’s crucial restricting the access to the bucket; create an unprivileged IAM user as follows:

aws iam create-user --user-name terraform-deployer
# Output:
{
    "User": {
        "UserName": "terraform-deployer",
        "Path": "/",
        "CreateDate": "2019-01-27T03:20:41.270Z",
        "UserId": "AIDAIOSFODNN7EXAMPLE",
        "Arn": "arn:aws:iam::123456789012:user/terraform-deployer"
    }
}

Take note of the Arn from the command’s output (it looks like: “Arn”: “arn:aws:iam::123456789012:user/terraform-deployer”).

To correctly interact with the s3 service and DynamoDB at a later stage to implement the locking, our IAM user must hold a sufficient set of permissions. It is recommended to have severe restrictions in place for production environments, though, for the sake of simplicity, start assigning AmazonS3FullAccess and AmazonDynamoDBFullAccess:

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --user-name terraform-deployer
# Output: expect none when the command execution is successful

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess --user-name terraform-deployer
# Output: expect none when the command execution is successful

The freshly created IAM user must be enabled to execute the required actions against your s3 bucket. You can do this by creating and applying the right policy, as follows:

cat <<-EOF >> policy.json
{
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/terraform-deployer"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::terraform-remote-store"
        }
    ]
}
EOF

This basic policy file grants the principal with arn “arn:aws:iam::123456789012:user/terraform-deployer”, to execute all the available actions (“Action”: “s3:*") against the bucket with arn “arn:aws:s3:::terraform-remote-store”. Again, in production is desired to force way stricter policies. For reference, have a look at the AWS Policy Generator.

Back to the terminal and run the command as shown below, to enforce the policy in your bucket:

aws s3api put-bucket-policy --bucket terraform-remote-store --policy file://policy.json
# Output: none

As the last step, enable the bucket’s versioning:

aws s3api put-bucket-versioning --bucket terraform-remote-store --versioning-configuration Status=Enabled

It allows saving different versions of the infrastructure’s state and rollback easily to a previous stage without struggling.

The AWS s3 bucket is ready, time to integrate it with Terraform. Listed below, is the minimal configuration required to set up this remote backend:

# terraform.tf

provider "aws" {
  region                  = "${var.aws_region}"
  shared_credentials_file = "~/.aws/credentials"
  profile                 = "default"
}

terraform {  
    backend "s3" {
        bucket  = "terraform-remote-store"
        encrypt = true
        key     = "terraform.tfstate"    
        region  = "eu-west-1"  
    }
}

# the rest of your configuration and resources to deploy

Once in place, terraform must be initialized (again). terraform init The remote backend is ready for a ride, test it.

What about locking? Storing the state remotely brings a pitfall, especially when working in scenarios where several tasks, jobs, and team members have access to it. Under these circumstances, the risk of multiple concurrent attempts to make changes to the state is high. Here comes to help the lock, a feature that prevents opening the state file while already in use.

You can implement the lock creating an AWS DynamoDB Table, used by terraform to set and unset the locks. Provision the resource using terraform itself:

# create-dynamodb-lock-table.tf
resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {
  name           = "terraform-state-lock-dynamo"
  hash_key       = "LockID"
  read_capacity  = 20
  write_capacity = 20
attribute {
    name = "LockID"
    type = "S"
  }
tags {
    Name = "DynamoDB Terraform State Lock Table"
  }
}

and deploy it as shown: terraform plan -out "planfile" && terraform apply -input=false -auto-approve "planfile"

Once the command execution is completed, the locking mechanism must be added to your backend configuration as follow:

# terraform.tf

provider "aws" {
  region                  = "${var.aws_region}"
  shared_credentials_file = "~/.aws/credentials"
  profile                 = "default"
}

terraform {  
    backend "s3" {
        bucket         = "terraform-remote-store"
        encrypt        = true
        key            = "terraform.tfstate"    
        region         = "eu-west-1"
        dynamodb_table = "terraform-state-lock-dynamo"
    }
}

# the rest of your configuration and resources to deploy

All done. Remember to run again terraform init and enjoy your remote backend.

View more solutions

Comments

[Oliver Charlesworth]

This is a good question. FWIW we had a separate "bootstrap" TF project, which in turn relied on a super-minimal manually provisioned bucket.

[ydaetskcoR]

Yeah I've done something similar where a bootstrap project copies across a bunch of helper scripts and provider configs for a project and also creates a versioned S3 bucket and DynamoDb lock table if it doesn't exist using the AWS CLI. It would be nice if we could do that in Terraform but when I tried it was too messy to be worth it.

[Begin]

Terragrunt also can take care of this for you, and makes it really convenient if you want to move to a different bucket. github.com/gruntwork-io/terragrunt

[Alapati]

Is there a better way to do this in 2020 ? I have seen suggestions of using a local state for the s3 creation in a different folder. I don't think that's the right approach for a CD plan. Have anyone found a better way ?

[tomarv2]

I used the same solution that you suggested, I have a small project that I use to manage remote state you can see here: github.com/tomarv2/tfremote

[Jepper]

Brilliant answer. FWIW anyone reading "Terraform Up and Running" and stuck in chapter 3 backends (which is using an older version of terreform) this is the solution.

[AndrewKS]

I assume you're using the terraform_state_lock table as a mutex for writing the state, but what reads/writes from it? Does terraform do something behind the scenes there? What happens if you exclude the aws_dynamodb_table resource?

[AndrewKS]

Ah -- I figured it out. It's because the built-in s3 backend supports locking via a dynamodb table using the dynamodb_table param: terraform.io/docs/backends/types/s3.html#dynamodb_table

[Andriy Drozdyuk]

What do you mean by "remote-state folder"?

[RichVel]

@drozzy - not the author, but "remote state folder" in last para means the folder that contains this .tf file, which should be a sub-folder in your repo, say remote-state. So you will need to do cd remote-state before those two Terraform commands. I will suggest an edit to clarify.

[RichVel]

Actually, you can use Terraform to build the remote state components (S3 bucket and DynamoDB table) - just use a separate sub-folder for building these, which has its own (local) Terraform state file. See this answer.

[allan.simon]

just to be sure: so you have 2 folder dev/remote-state and prod/remote-state , which point to 2 buckets s3 "dev-tfstate-bucket" and "prod-tfstate-bucket" respectively ?

[akskap]

I am still not able to understand the answer completely.. When we cd remote-state and run terraform apply - terraform.state for this base infra is still locally saved (with no backend configured), right ? How does this solve the chicken/egg problem ? Does this state file also come under purview of terraform management now ?

[tdensmore]

@akskap it does not COMPLETELY solve the chicken and egg problem, but the state complexity of creating one bucket to bootstrap the rest of the system is minimal. The state can be easily imported if necessary, so I think it is easy to ignore this issue in order to implement this trivial solution. WARNING: since the projects are now split-brained, the bucket names must match or you might run into misleading access denied errors (trying to list resources in other peoples buckets).

[utkarsh867]

I've run into a problem while following this process. I cannot create the dynamo db using terraform if I want to frequently destroy other resources during development. Any suggestions to deal with that issue, or should I just use the CLI?

[Dmitry Kutetsky]

Agree, this is a preety simple and useful way for the initial setup.

[Marcello de Sales]

@utkarsh867 just split the setup of S3 and Dynamo in an isolated project... That way, you can deal with your other resources during development... I personally created a Docker image to setup S3 (shell and terraform itself)...

[AndrewKS]

@akskap There's always going to be a chicken-egg problem, but the solution with this answer is that you isolate that chicken-egg problem to a very small, likely unchanging terraform state for the S3 bucket. This removes the chicken-egg problem for your potentially large, always changing state of your total infrastructure.

[Scott Stensland]

this errors out │ Error: Value for unconfigurable attribute see open ticket github.com/hashicorp/terraform-provider-aws/issues/23106

[Scott Stensland]

errors out ` Error: Value for unconfigurable attribute ` with Can't configure a value for "versioning": its value will be decided automatically based on the result of applying this configuration.

[Orkhan M.]

Hi, it seems that you mistyped bucket name when you were showing the way to create policies.