Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creation of prj-c-secrets fails with error "already exists" #1187

Closed
mromascanu123 opened this issue Apr 11, 2024 · 4 comments
Closed

Creation of prj-c-secrets fails with error "already exists" #1187

mromascanu123 opened this issue Apr 11, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@mromascanu123
Copy link

TL;DR

Creation of the prj-c-secrets project (in 1-org package) fails with "already exists" issue. Not clear what is "already existing" because the project ID at stake ("prj-c-secrets-zfzs") does not exist
gcloud projects list | grep prj-c-secret
prj-c-secrets-1q2n prj-c-secrets 560518446142
prj-c-secrets-bb5k prj-c-secrets 536016672477
prj-c-secrets-cb7v prj-c-secrets 298129689217

Possibly a red-herring because just before this error there is a reference to a null project-id

module.org_secrets.module.project-factory.google_project.main: Creating...
2024-04-10T23:56:30.305Z [INFO] Starting apply for module.org_secrets.module.project-factory.google_project.main
2024-04-10T23:56:30.305Z [DEBUG] module.org_secrets.module.project-factory.google_project.main: applying the planned Create change
... etc...

2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Request Details:
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]---------------------------------------
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: GET /v1/projects/00000000000/services/serviceusage.googleapis.com?alt=json&prettyPrint=false HTTP/1.1
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: serviceusage.googleapis.com
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: -----------------------------------------------------
2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details:
2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]--------------------------------------
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 403 Forbidden
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 403,
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5z__k",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5z__k",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "forbidden"
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ],
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "PERMISSION_DENIED",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "details": [
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "violations": [
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "googleapis.com",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "subject": "?error_code=210002&type=Project&resource_id=00000000000"
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ]
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: },
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.ErrorInfo",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "RESOURCES_NOT_FOUND",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "serviceusage.googleapis.com",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "metadata": {
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "Project",
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "resource_id": "00000000000"
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ]
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }

Everything happens inside the attempted call to project-factory module in 1-org/envs/shared/projects.tf

/******************************************
Project for Org-wide Secrets
*****************************************/

module "org_secrets" {
source = "terraform-google-modules/project-factory/google"

random_project_id = true
random_project_id_length = 4
default_service_account = "deprivilege"
name = "${local.project_prefix}-c-secrets"
org_id = local.org_id
billing_account = local.billing_account
folder_id = google_folder.common.id
activate_apis = ["logging.googleapis.com", "secretmanager.googleapis.com", "billingbudgets.googleapis.com"]

labels = {
environment = "production"
application_name = "org-secrets"
billing_code = "1234"
primary_contact = "example1"
secondary_contact = "example2"
business_code = "abcd"
env_code = "p"
}
budget_alert_pubsub_topic = var.project_budget.org_secrets_alert_pubsub_topic
budget_alert_spent_percents = var.project_budget.org_secrets_alert_spent_percents
budget_amount = var.project_budget.org_secrets_budget_amount
budget_alert_spend_basis = var.project_budget.org_secrets_budget_alert_spend_basis
}

Terraform log attached

Expected behavior

As all the other projects have been created successfully in 1-org, all using project-factory (see attached screenshot) , so should be prj-c-secrets

Observed behavior

Creation fails and the NULL strange project ID might be related to issue 1186

Terraform Configuration

Nothing special - see attached screenshot

Terraform Version

$ terraform version
Terraform v1.6.0
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.8.0.

Additional information

Here is a log extract:
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]---------------------------------------
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: POST /v1/projects?alt=json&prettyPrint=false HTTP/1.1
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: cloudresourcemanager.googleapis.com
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Length: 296
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "labels": {
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "application_name": "org-secrets",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "billing_code": "1234",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "business_code": "abcd",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "env_code": "p",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "environment": "production",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "primary_contact": "example1",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "secondary_contact": "example2"
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: },
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "name": "prj-c-secrets",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "parent": {
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "id": "384831136297",
2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "folder"
2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: },
2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "projectId": "prj-c-secrets-zfzs"
2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: -----------------------------------------------------
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details:
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]--------------------------------------
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 409 Conflict
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server-Timing: gfet4t7; dur=195
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": {
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 409,
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists",
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: {
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists",
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global",
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "alreadyExists"
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ],
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "ALREADY_EXISTS"
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5

terraform_debug.log

other_org_projects_created

terraform_tfvars
@mromascanu123 mromascanu123 added the bug Something isn't working label Apr 11, 2024
@mromascanu123
Copy link
Author

Here is the funny thing - to "fix" (kind of) the issue simply bumped random_project_id_length from 4 to 6 in module org_secrets in 1-org/envs/shared/projects.tf

Does not mean this is no longer a bug but I don't have an explanation - in the tfstate file the resource "random_string" "random_project_id_suffix" was existing and as a result there was no real randomization of the project_id between subsequent plan & apply operations. Once I bumped the hength here you go:

module.org_secrets.module.project-factory.random_string.random_project_id_suffix[0] must be replaced

-/+ resource "random_string" "random_project_id_suffix" {
~ id = "zfzs" -> (known after apply)
~ length = 4 -> 6 # forces replacement
~ result = "zfzs" -> (known after apply)
# (9 unchanged attributes hidden)
}

@mromascanu123 mromascanu123 mentioned this issue Apr 12, 2024
Closed
@eeaton
Copy link
Collaborator

eeaton commented Apr 24, 2024

This is a terraform pain point when you're creating GCP project IDs with a random suffix. This happens when a terraform apply step fails midway through: your terraform state has generated the name of a project ID, but the project has not been created. This causes subsequent attempts to use terraform plan or terraform apply to fail because they're looking for a project ID that does not exist.

This error is not unique to the code in this repo, but it is an error you're likely to encounter because this repo creates a large number of projects, and a failed apply step due to project quota limitations often leads to this error.

error messages

You might also see errors like this (including these for others who might be searching for the specific error text):

converting TF resource to CAI: getting resource ancestry or parent failed: user does not have the correct permissions for projects/prj-c-secrets-zfzs. For more info: https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden]. Additional details: [terraform-validator-internal.git.corp.google.com/terraform-tools.git/cmd.Execute

Or errors like this:

Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid

fix

Unfortunately we have to unpick terraform state to resolve the error. The terraform destroy command doesn't help because it will fail with the same error trying to find the invalid project.

Your workaround to force replacement of the random suffix by changing it's length works because it gets rid of the generated suffix in state.

I prefer to use the Terraform cli to remove the suffix from state, then run again

terraform plan 
# returns error like Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid

terraform state list
# look for the state object ending in random_project_id_suffix[0]. Some steps have a few of these, choose the one aligned to the module for the project in the error. 

terraform state show module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]
# check that you choose the correct suffix as the project in the error, this should contain id = "zfzs" (or the same value as your suffix) 

# After you have verified this is the correct ID, remove it.
terraform state rm module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]

terraform plan
# it should work now

I'll create an issue to improve readme guidance on fixing this error state because it's non-intuitive to workaround.

@eeaton eeaton mentioned this issue Apr 24, 2024
Closed
@eeaton
Copy link
Collaborator

eeaton commented Apr 24, 2024

I will track this in #1195

@eeaton eeaton closed this as completed Apr 24, 2024
@fmichaelobrien
Copy link
Contributor

Sounds good will track 1195 in GoogleCloudPlatform/pbmm-on-gcp-onboarding#380

@eeaton eeaton mentioned this issue May 20, 2024
Closed
@eeaton eeaton mentioned this issue Jun 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@eeaton @fmichaelobrien @mromascanu123