Google Cloud credentials with Terraform

Question

This is a bit of a newbie question, but I've just gotten started with GCP provisioning using Terraform / Terragrunt, and I find the workflow with obtaining GCP credentials quite confusing. I've come from using AWS exclusively, where obtaining credentials, and configuring them in the AWS CLI was quite straightforward.

Basically, the Google Cloud Provider documentation states that you should define a provider block like so:

provider "google" {
  credentials = "${file("account.json")}"
  project     = "my-project-id"
  region      = "us-central1"
  zone        = "us-central1-c"
}

This credentials field shows I (apparently) must generate a service account, and keep a JSON somewhere on my filesystem.

However, if I run the command gcloud auth application-default login, this generates a token located at ~/.config/gcloud/application_default_credentials.json; alternatively I can also use gcloud auth login <my-username>. From there I can access the Google API (which is what Terraform is doing under the hood as well) from the command line using a gcloud command.

So why does the Terraform provider require a JSON file of a service account? Why can't it just use the credentials that the gcloud CLI tool is already using?

By the way, if I configure Terraform to point to the application_default_credentials.json file, I get the following errors:

Initializing modules...

Initializing the backend...

Error: Failed to get existing workspaces: querying Cloud Storage failed: Get https://www.googleapis.com/storage/v1/b/terraform-state-bucket/o?alt=json&delimiter=%2F&pageToken=&prefix=projects%2Fsomeproject%2F&prettyPrint=false&projection=full&versions=false: private key should be a PEM or plain PKCS1 or PKCS8; parse error: asn1: syntax error: sequence truncated

Solution

if I configure Terraform to point to the application_default_credentials.json file, I get the following errors:

The credentials field in provider config expects a path to service account key file, not user account credentials file. If you want to authenticate with your user account try omitting credentials and then running gcloud auth application-default login; if Terraform doesn't find your credentials file you can set the GOOGLE_APPLICATION_CREDENTIALS environment variabe to point to ~/.config/gcloud/application_default_credentials.json.

Read here for more on the topic of service accounts vs user accounts. For what it's worth, Terraform docs explicitly advice against using application-default login:

This approach isn't recommended- some APIs are not compatible with credentials obtained through gcloud

Similarly GCP docs state the following:

Important: For almost all cases, whether you are developing locally or in a production application, you should use service accounts, rather than user accounts or API keys.

Comments

[Scott Crooks]

Ah, I see now, thank you very much! I was so confused about why this happens. The part that you mention from the Terraform Docs ("This approach isn't recommended- some APIs are not compatible with credentials obtained through gcloud") still worries me a bit. That implies that some actions I cannot do through a regular gcloud command. That seems very strange.

[Scott Crooks]

actually it looks like a user must use a service account. When I do your recommended steps, and point GOOGLE_APPLICATION_CREDENTIALS to ~/.config/gcloud/application_default_credentials.json, it still gives me the error above, saying that it's not in PEM format.

[Aleksi]

Oh right, my bad. Does application-default login still work if you unset the GOOGLE_APPLICATION_CREDENTIALS variable?

[Matt Drees]

FWIW, the linked 'GCP Docs' no longer contain the 'you should use service accounts' phrase. User accounts don't seem to be discouraged, and appear to be intended for developers.

[cryanbhu]

this is wrong, the same link, registry.terraform.io/providers/hashicorp/google/latest/docs‌​/… states: If you are using Terraform on your workstation we recommend that you install gcloud and authenticate using User Application Default Credentials ("ADCs") as a primary authentication method. You can enable ADCs by running the command gcloud auth application-default login.