1-org: Getting resource ancestry or parent failed: user does not have the correct permissions

[lpezet]

TL;DR

When running ./tf-wrapper.sh plan_validate_all as part of the Github tf-pull-request after creating PR, the following error is raised:

ERROR: [module.base_restricted_environment_network["nonproduction"].module.base_shared_vpc_host_project.module.project-factory.google_service_account.default_service_account[0]: converting TF resource to CAI: getting resource ancestry or parent failed: user does not have the correct permissions for projects/prj-n-shared-base-xxxx. For more info: https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden]. Additional details: [terraform-validator-internal.git.corp.google.com/terraform-tools.git/cmd.Execute
	/tmpfs/src/git/terraform-tools/cmd/root.go:93
main.main
	/tmpfs/src/git/terraform-tools/main.go:16
runtime.main
	/usr/local/go/src/runtime/proc.go:250]
DEBUG: Chosen display Format:default
INFO: Display format: "default"
DEBUG: (gcloud.beta.terraform.vet) 
Traceback (most recent call last):
  File "/opt/hostedtoolcache/gcloud/486.0.0/x64/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
    resources = calliope_command.Run(cli=self, args=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/gcloud/486.0.0/x64/lib/googlecloudsdk/calliope/backend.py", line 828, in Run
    raise exceptions.ExitCodeNoError(exit_code=command_instance.exit_code)
googlecloudsdk.calliope.exceptions.ExitCodeNoError
Error: Process completed with exit code 33.

Expected behavior

No errors

Observed behavior

Command and full output (loosely obfuscated):

$ gcloud beta terraform vet "./tmp_plan/envs-shared.json" 
--policy-library="./policy-library/" --project="prj-b-cicd-wif-gh-xxxx" --verbosity=debug --impersonate-service-accou
[email protected]
DEBUG: Running [gcloud.beta.terraform.vet] with arguments: [--impersonate-service-account: "[email protected]", --policy-library: "./policy-library/", --project: "prj-b-cicd-wif-gh-xxxx", --verbosity: "debug", TERRAFORM_PLAN_JSON: "./tmp_plan/envs-shared.json"]
WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]].
DEBUG: Making request: POST https://oauth2.googleapis.com/token
DEBUG: Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG: https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG: Making request: POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken
DEBUG: Starting new HTTPS connection (1): iamcredentials.googleapis.com:443
DEBUG: https://iamcredentials.googleapis.com:443 "POST /v1/projects/-/serviceAccounts/[email protected]:generateAccessToken HTTP/1.1" 200 None
DEBUG: Setting project to prj-b-cicd-wif-gh-xxxx from properties
DEBUG: Executing command: ['/usr/lib/google-cloud-sdk/bin/terraform-tools', 'tfplan-to-cai', './tmp_plan/envs-shared.json', '--output-path', '/tmp/tmpk1gedqb1/cai_assets.json', '--verbosity', 'debug', '--user-agent', 'CloudSDK/485.0.0 (Linux 5.15.146.1-microsoft-standard-WSL2)', '--project', 'prj-b-cicd-wif-gh-xxxx']
INFO: [[INFO] Authenticating using configured Google JSON 'access_token'...].
INFO: [[INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]].
INFO: [[INFO] Authenticating using configured Google JSON 'access_token'...].
INFO: [[INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]].
INFO: [[DEBUG] Waiting for state to become: [success]].
INFO: [[INFO] Terraform is using this identity: [email protected]].
INFO: [[INFO] Instantiating Google Cloud ResourceManager client for path https://cloudresourcemanager.googleapis.com/].
INFO: [[INFO] Instantiating Google Cloud ResourceManager V3 client for path https://cloudresourcemanager.googleapis.com/].
INFO: [[INFO] Instantiating Google Storage client for path https://storage.googleapis.com/storage/v1/].
DEBUG: [google_essential_contacts_contact.essential_contacts["[email protected]"]: resource type cannot be converted for CAI-based policies: google_essential_contacts_contact. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_essential_contacts_contact.essential_contacts["[email protected]"]: resource type cannot be converted for CAI-based policies: google_essential_contacts_contact. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_essential_contacts_contact.essential_contacts["[email protected]"]: resource type cannot be converted for CAI-based policies: google_essential_contacts_contact. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_essential_contacts_contact.essential_contacts["[email protected]"]: resource type cannot be converted for CAI-based policies: google_essential_contacts_contact. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_folder_iam_audit_config.folder_config[0]: resource type cannot be converted for CAI-based policies: google_folder_iam_audit_config. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_binding.bootstrap_folder: resource type cannot be converted for CAI-based policies: google_tags_tag_binding. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_binding.common_folder: resource type cannot be converted for CAI-based policies: google_tags_tag_binding. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_binding.network_folder: resource type cannot be converted for CAI-based policies: google_tags_tag_binding. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_key.tag_keys["environment"]: resource type cannot be converted for CAI-based policies: google_tags_tag_key. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_value.tag_values["environment_bootstrap"]: resource type cannot be converted for CAI-based policies: google_tags_tag_value. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_value.tag_values["environment_development"]: resource type cannot be converted for CAI-based policies: google_tags_tag_value. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_value.tag_values["environment_nonproduction"]: resource type cannot be converted for CAI-based policies: google_tags_tag_value. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [google_tags_tag_value.tag_values["environment_production"]: resource type cannot be converted for CAI-based policies: google_tags_tag_value. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_domain_restricted_sharing.data.google_organization.orgs["example.com"]: resource type not found in google GA provider: google_organization.].
DEBUG: [module.common_kms.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.common_kms.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.dns_hub.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.dns_hub.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.interconnect.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.interconnect.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.destination_aggregated_logs[0].google_logging_linked_dataset.linked_dataset[0]: resource type cannot be converted for CAI-based policies: google_logging_linked_dataset. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.destination_aggregated_logs[0].google_logging_project_bucket_config.bucket: resource type cannot be converted for CAI-based policies: google_logging_project_bucket_config. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.internal_project_log_export[0].google_logging_project_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_project_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export["823309480423_prj"].google_logging_folder_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_folder_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export["823309480423_pub"].google_logging_folder_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_folder_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export["823309480423_sto"].google_logging_folder_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_folder_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export_billing["prj"].google_logging_billing_account_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_billing_account_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export_billing["pub"].google_logging_billing_account_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_billing_account_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.logs_export.module.log_export_billing["sto"].google_logging_billing_account_sink.sink[0]: resource type cannot be converted for CAI-based policies: google_logging_billing_account_sink. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_audit_logs.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_audit_logs.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_billing_export.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_billing_export.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_secrets.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.org_secrets.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.scc_notifications.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.scc_notifications.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["development"].module.base_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["development"].module.base_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["development"].module.restricted_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["development"].module.restricted_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["nonproduction"].module.base_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["nonproduction"].module.base_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["nonproduction"].module.restricted_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["nonproduction"].module.restricted_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["production"].module.base_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["production"].module.base_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["production"].module.restricted_shared_vpc_host_project.module.budget.google_billing_budget.budget[0]: resource type cannot be converted for CAI-based policies: google_billing_budget. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
DEBUG: [module.base_restricted_environment_network["production"].module.restricted_shared_vpc_host_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: resource type cannot be converted for CAI-based policies: google_project_default_service_accounts. For details, see https://cloud.google.com/docs/terraform/policy-validation/create-cai-constraints#supported_resources].
INFO: [[INFO] Instantiating Google Cloud ResourceManager client for path https://cloudresourcemanager.googleapis.com/].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=cloudresourcemanager.googleapis.com/Project)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [[DEBUG] matching ID tp-org-logs-xxxx to regex (?P<topic>[^/]+).].
INFO: [[DEBUG] Waiting for state to become: [success]].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=pubsub.googleapis.com/Topic)].
INFO: [[DEBUG] matching ID bkt-prj-c-logging-xxxx-org-logs-xxxx to regex (?P<bucket>[^/]+).].
INFO: [[DEBUG] Waiting for state to become: [success]].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=storage.googleapis.com/Bucket)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=cloudresourcemanager.googleapis.com/Folder)].
INFO: [Retrieving ancestry from resource (type=cloudresourcemanager.googleapis.com/Project)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=cloudbilling.googleapis.com/ProjectBillingInfo)].
INFO: [Retrieving ancestry from resource (type=iam.googleapis.com/ServiceAccount)].
INFO: [Retrieving ancestry from resource (type=cloudresourcemanager.googleapis.com/Project)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 403 with body: HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 31 Jul 2024 04:16:19 GMT
Server: ESF
Server-Timing: gfet4t7; dur=163
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "errors": [
      {
        "message": "The caller does not have permission",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request was successful].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=cloudbilling.googleapis.com/ProjectBillingInfo)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 403 with body: HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 31 Jul 2024 04:16:19 GMT
Server: ESF
Server-Timing: gfet4t7; dur=127
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "errors": [
      {
        "message": "The caller does not have permission",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
INFO: [Retrieving ancestry from resource (type=iam.googleapis.com/ServiceAccount)].
INFO: [[DEBUG] Retry Transport: starting RoundTrip retry loop].
INFO: [[DEBUG] Retry Transport: request attempt 0].
INFO: [[DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 403 with body: HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 31 Jul 2024 04:16:19 GMT
Server: ESF
Server-Timing: gfet4t7; dur=176
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "errors": [
      {
        "message": "The caller does not have permission",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}].
INFO: [[DEBUG] Retry Transport: Returning after 1 attempts].
ERROR: [module.base_restricted_environment_network["nonproduction"].module.base_shared_vpc_host_project.module.project-factory.google_service_account.default_service_account[0]: converting TF resource to CAI: getting resource ancestry or parent failed: user does not have the correct permissions for projects/prj-n-shared-base-xxxx. For more info: https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden]. Additional details: [terraform-validator-internal.git.corp.google.com/terraform-tools.git/cmd.Execute
        /tmpfs/src/git/terraform-tools/cmd/root.go:93
main.main
        /tmpfs/src/git/terraform-tools/main.go:16
runtime.main
        /usr/local/go/src/runtime/proc.go:250]
DEBUG: Chosen display Format:default
INFO: Display format: "default"
DEBUG: (gcloud.beta.terraform.vet) 
Traceback (most recent call last):
  File "/usr/bin/../lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
    resources = calliope_command.Run(cli=self, args=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/../lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 828, in Run
    raise exceptions.ExitCodeNoError(exit_code=command_instance.exit_code)
googlecloudsdk.calliope.exceptions.ExitCodeNoError

Terraform Configuration

Terraform Version

Same behavior using 2 different Terraform versions:
1.5.7/linux_amd64 - Github Action hashicorp/setup-terraform@v2 (from terraform-example-foundation github workflow file)
1.9.2/linux_amd64 - locally

Additional information

As mentioned in (provided link from error message)[https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden], I added --verbosity=debug to find the identity used during the gcloud beta terraform vet call in the tf-wrapper.sh script.
Problem is that in Github will obfuscate that kind of thing and I end up with:

INFO: [[DEBUG] Waiting for state to become: [success]].
INFO: [[INFO] Terraform is using this identity: ***].
INFO: [[INFO] Instantiating Google Cloud ResourceManager client for path https://cloudresourcemanager.googleapis.com/].

I then ran (what I believe is) the same gcloud beta terraform vet command using the [email protected] but I get the same error.

I noticed 2 (much) earlier issues somewhat related to this permission issue: #620 and #546

[eeaton]

In the Observed Behavior section, the same error shows up for every resource type under Debug logs, which makes me suspect that Auth is misconfigured. I'm not able to reproduce in our CI pipelines, which successfully pass the plan and validate stages in from tf-wrapper.sh.

Is this a persistent blocker, or transient?

[daniel-cit]

The [email protected] service account should have these roles in the Organization:

• Access Context Manager Admin
• Browser
• Cloud Asset Owner
• Essential Contacts Admin
• Logs Configuration Writer
• Organization Administrator
• Security Center Notification Configurations Editor
• Security Center Sources Editor
• Tag Administrator
• Tag User

The role Browser has the two permissions needed to access the project

• resourcemanager.projects.getIamPolicy
• resourcemanager.projects.get

Cloud you please check if the Service Account has the correct roles?

[lpezet]

@eeaton It's a persistent blocker.
@daniel-cit My [email protected] has those roles (and even an extra one: Storage Object Admin).
I'll destroy everything and start from scratch (once quota issue resolved). Thanks for the help!

[adamcox-acquired]

Hi all, I think I may be able to shed some light on the problem here.

I've just had almost the exact same thing happen while adding a new project in this stage, and after a few hours of banging my head against it I managed to resolve it.

The issue occurred for me when there had been a previously failed apply (I ran out of Project Quota). I was using the project-factory module with the random_project_id setting enabled. In the first apply, it managed to create only the module.my_new_project.module.project-factory.random_id.random_project_id_suffix and module.my_new_project.module.project-factory.random_string.random_project_id_suffix[0] resources, and then failed due to the quota issue when attempting to create the project itself.

After getting the quota expanded I attempted to re-run the GitHub pipeline and received this same type of error, directed at the module.my_new_project.module.project-factory.google_service_account.default_service_account[0] resource. No amount of validating and/or increasing permissions made any difference.

Eventually I noticed that the random_id/random_string resources had been created and stored in the state file, and because of that Terraform now knew the final name of the project. On a hunch I ran terraform destroy -target module.my_new_project to remove only the two created resources. After that I was able to run a full plan/validate/apply cycle with no further issues.

Hopefully this detail will aid in finding the root cause! 🤞