"code: 403, message: Authentication failure" when using the provider

[Linuxine]

HI,

I am sorry if this is an obvious question, but I did not manage to find any answer in this provider or GGP documentation. I am trying to use this provider to create a storage spool and some volumes on GCP.

I created a Kubernetes Cluster with terraform, logged in as a user via the gcloud client, without any issue. My user is "Owner" on my project, since this is a dev project, I did not care for excessive rights.

As I understood, to use the netapp gcp provider, I needed to create a service account on my project. So I created one, and gave him the NetApp Cloud Volumes Admin Role. I made sure my gcloud user can impersonate this service account by adding the Service Account Token Creator to my user, and so I can now create storage pools and volumes with gcloud commands, using the --impersonate-service-account option.

But when I try to use this service account to create pools and volumes via terraform, it fails with this error (here in debug mode) :

2023-11-13T14:48:18.320+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] Waiting for state to become: [success]
2023-11-13T14:48:18.412+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [INFO] Terraform is using this identity: [email protected]
2023-11-13T14:48:18.412+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] parent context canceled, cleaning up batcher batches
2023-11-13T14:48:18.412+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] Stopping batcher "Service Usage"
2023-11-13T14:48:18.412+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] parent context canceled, cleaning up batcher batches
2023-11-13T14:48:18.412+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] Stopping batcher "IAM"
2023-11-13T14:48:18.414+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] [transport] [server-transport 0xc000d3c1a0] Closing: Server.Stop called
2023-11-13T14:48:18.414+0100 [DEBUG] provider.terraform-provider-google_v5.5.0_x5: 2023/11/13 14:48:18 [DEBUG] [transport] [server-transport 0xc000d3c1a0] loopyWriter exiting with error: transport closed byclient
2023-11-13T14:48:18.414+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-13T14:48:18.419+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.5.0/linux_amd64/terraform-provider-google_v5.5.0_x5 pid=39233
2023-11-13T14:48:18.419+0100 [DEBUG] provider: plugin exited
2023-11-13T14:48:18.715+0100 [DEBUG] provider.terraform-provider-netapp-gcp_v23.4.0: 2023/11/13 14:48:18 createStoragePool request failed
2023-11-13T14:48:18.715+0100 [DEBUG] provider.terraform-provider-netapp-gcp_v23.4.0: 2023/11/13 14:48:18 Error creating storage pool: &errors.errorString{s:"code: 403, message: Authentication failure"}
2023-11-13T14:48:18.736+0100 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2023-11-13T14:48:18.736+0100 [ERROR] vertex "netapp-gcp_storage_pool.regional-storage-pool" error: code: 403, message: Authentication failure
╷
│ Error: code: 403, message: Authentication failure
│
│   with netapp-gcp_storage_pool.regional-storage-pool,
│   on netapp_volume.tf line 19, in resource "netapp-gcp_storage_pool" "regional-storage-pool":
│   19: resource "netapp-gcp_storage_pool" "regional-storage-pool" {
│
╵
2023-11-13T14:48:18.755+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-13T14:48:18.760+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/netapp/netapp-gcp/23.4.0/linux_amd64/terraform-provider-netapp-gcp_v23.4.0 pid=39248
2023-11-13T14:48:18.760+0100 [DEBUG] provider: plugin exited

My terraform file is like this:

terraform {
  required_providers {

    netapp-gcp = {
      source = "NetApp/netapp-gcp"
      version = "~> 23.4.0"
    }
  }

  required_version = "~> 1.3"
}

provider "netapp-gcp" {
  project         = "xxx"
  #service_account = "../xxx.json"
  service_account = "[email protected]"
}

# local variables
locals {
  network = var.network
  pool_name = "project-storage-pool"
  region = "europe-west4"
  zone  = "europe-west4-a"
  secondary_zone = "europe-west4-b"
  pool_size = 2048
}

resource "netapp-gcp_storage_pool" "nfs_cvs_sp" {
  name           =  local.pool_name
  region         = var.region
  zone           = local.zone
  secondary_zone = local.secondary_zone
  network        = var.network
  size           = local.pool_size
  service_level  = "ZoneRedundantStandardSW"
  storage_class  = "software"
}

As you can see commented out, I tried to provide the path to the service account keys json file, or the name of the service account, with the same result.

I am guessing there is something wrong with the way I defined my service account ? But it is strange that it seems to be ok when using the gcloud client.
Thanks a lot in advance for any help !

[okrause]

Sounds like you did everything right to setup service account and impersonation.
One common gotcha: Like all APi code, it is using Googles Application Default Credentials (ADC). A common misconception is that gcloud sets them automatically when doing gcloud auth login. That's not the case. You need to generate the using gcloud auth application-default login. Run that and have another try.

[Linuxine]

Hi @okrause , thanks a lot for your answer !
Indeed, I had performed a gcloud auth login but not a gcloud auth application-default login. Sadly, though, I still got the same error after running this command.

I tried to perform a gcloud projects get-iam-policy <project> to check if my service account has the right roles, and I got the result:

- members:
  - serviceAccount:[email protected]
  role: roles/editor

- members:
  - serviceAccount:[email protected]
  - user:<my gcloud user>
  role: roles/netapp.admin

- members:
  - serviceAccount:[email protected]
  - user:<my gcloud user>
  role: roles/netappcloudvolumes.admin

- members:
  - user:<my gcloud user>
  role: roles/iam.serviceAccountTokenCreator

I was not sure if the right role to set was "NetApp Cloud Volumes Admin" or "Google Cloud NetApp Volumes Admin", so I added both in the GUI, could this be an issue ? I also added the right "editor" to the service account to make sure it was not missing any permissions, but it does not seems to help. Is there any other command I could try to find what could be the issue ?

Thanks again !

[okrause]

role: roles/netappcloudvolumes.admin is the right one. You are using Cloud NetApp Volume, correct? Check with:

gcloud services list --filter="name:cloudvolumesgcp-api.netapp.com" NAME TITLE cloudvolumesgcp-api.netapp.com NetApp Cloud Volumes API
But I thing I see the issue:
You granted your user roles/iam.serviceAccountTokenCreator. Please read the instructions how to configure service account impersonation here: https://cloud.google.com/architecture/partners/netapp-cloud-volumes/api#authenticate_using_service_account_impersonation

You need to grant roles/iam.serviceAccountTokenCreator to your user on the service account.
Check with gcloud iam service-accounts get-iam-policy [email protected]. It needs to show something like:

bindings:
- members:
  - user:<my gcloud user>
  role: roles/iam.serviceAccountTokenCreator

[Linuxine]

Indeed, I had forgotten to perform the add-iam-policy-binding between my user and the service account. I have now the binding :

❯ gcloud iam service-accounts get-iam-policy [email protected]
bindings:
- members:
  - user:<my gcloud user>
  role: roles/iam.serviceAccountTokenCreator
etag: BwYKG7R4dbY=
version: 1

But I still get the same authentication error... (I try after performing the  gcloud auth application-default login again, to make sure it has not expired).

Just to make sure, I am trying to create Google Cloud NetApp Volumes, directly in GCP, without using the NetApp Cloud Volumes Service. Am I right when I assume this is possible with this provider ? Thanks again !