Error creating vault key in azure

[jribmartins]

What happened?

I'm using the official azure provider to create a simple vault with some key.
I got always this:
"A resource with the ID xxx already exists - to be managed via Terraform this resource needs to be imported into the State."
This vault is being created also by me, has no keys, and I random generate names that's why I don't understand the error of the existing key.

How can we reproduce it?

Just apply the file attached, you just need to update tenantId and objectId values
reproduce.yaml.zip

What environment did it happen in?

• Crossplane Version: v1.12.2
• Provider Version: v0.33.0
• Kubernetes Version: v1.26.4+k3s1
• Kubernetes Distribution: k3d

[turkenf]

Hi @jribmartins,

Thank you for raising this issue. I was not able to reproduce the issue using the example you provided. Resources created successfully, I just removed providerConfigRef and updated tenantId, objectId parameters.

NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/test   True    True     test-ft2        23m

NAME                                       READY   SYNCED   EXTERNAL-NAME   AGE
key.keyvault.azure.upbound.io/test-vault   True    True     test-ft1        23m

NAME                                         READY   SYNCED   EXTERNAL-NAME   AGE
vault.keyvault.azure.upbound.io/test-vault   True    True     test-ft         23m

NAME                                                   READY   SYNCED   EXTERNAL-NAME                          AGE
accesspolicy.keyvault.azure.upbound.io/test-vault-sp   True    True     6c6f5c22-ff5b-4890-ba29-c5def658d16a   23m

Could it be a situation specific to your environment?

[jribmartins]

Hi @turkenf,
thanks for reviewing this.
It could be, but I don't find anything that can be specific from my environment, I'm trying to create a new vault with a new key.
I don't understand the error message saying that this resource already exists.

apply failed: A resource with the ID "https://atestcbs3.vault.azure.net/keys/atestcbs3/9aa8c5da418348a891976705b7300446" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_key_vault_key" for more information.:

I don't have other keyvault or other keyvaultkey with the name "atestcbs3" on my subscription

[turkenf]

@jribmartins could you please try after removing crossplane.io/external-name annotations or try different names for external-name?

[jribmartins]

Hi @turkenf I just tried, I got the same error, I'm generating random names with helm for the external-name, without success :/

[jeremypng]

I am having a similar issue. I create a new vault and new secrets and it is all synced. Then after a few hours, all of the secrets are out of sync and say that the object already exists and needs to be imported.

crossplane 1.13.2
provider-azure 0.36.0

[jeremypng]

In our case, after 1 hour, the secrets go ready false. I can reproduce it at will if there are any logs that would be helpful.

[djeremiah]

Digging into this a bit, it looks the id contains the version, which is a computed element. Right now, the controller is trying to pull the version information from the object spec, but the value isn't available until after the first Observe, so creation succeeds, but the resource immediately goes to unhealthy after the first reconciliation.

The solution is to switch the resources to use IdentifierFromProvider and capture the full resource id in the external-name. This will move control of the key name from the external-name annotation to spec.forProvider.name.

[ytsarev]

I can reproduce it on my side too

    Message:               apply failed: A resource with the ID "https://testvaultyury.vault.azure.net/keys/test/c5db10c53399413396fe1c5dccfa2d9c" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_key_vault_key" for more information.:

The associated tf state within the pod is empty

cat 04686c4b-4094-467f-a2ec-f87d252cdf7d/terraform.tfstate
{
  "version": 4,
  "terraform_version": "1.5.5",
  "serial": 2,
  "lineage": "04686c4b-4094-467f-a2ec-f87d252cdf7d",
  "outputs": {},
  "resources": [],
  "check_results": null
}

As @djeremiah hinted, the main suspect is keyVaultURLIDConf function in https://github.com/upbound/provider-azure/blob/main/config/externalname.go#L1861.

[ytsarev]

I created the draft PR with the tentative fix that looks promising

 k get -f reproduce.yaml
NAME                                              READY   SYNCED   EXTERNAL-NAME    AGE
vault.keyvault.azure.upbound.io/test-vault-yury   True    True     testvaultyury2   7m46s

NAME                                        READY   SYNCED   EXTERNAL-NAME                      AGE
key.keyvault.azure.upbound.io/test-vault1   True    True     066709f9668145b3a4886ee7a869f298   7m46s
key.keyvault.azure.upbound.io/test-vault2   True    True     067c7f951660486e812482e55452b007   7m46s
key.keyvault.azure.upbound.io/test-vault3   True    True     6549c187d5c94c4eae31aa4b344dc482   7m46s
key.keyvault.azure.upbound.io/test-vault4   True    True     198cdd4692c54d429a28ac111687227d   7m46s
key.keyvault.azure.upbound.io/test-vault5   True    True     0dce9ddb567348dbb1395eaaec32af98   7m46s

NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/test   True    True     test            7m46s

NAME                                                   READY   SYNCED   EXTERNAL-NAME                          AGE
accesspolicy.keyvault.azure.upbound.io/test-vault-sp   True    True     a721469d-bdb1-4b57-8916-cb0fbd7cbf7c   7m46s

[ytsarev]

Okay, something is super strange, I've noticed the inconsistent API behavior and rechecked the current code with the totally new vault. It also works.

k get -f reproduce.yaml
NAME                                              READY   SYNCED   EXTERNAL-NAME    AGE
vault.keyvault.azure.upbound.io/test-vault-yury   True    True     testvaultyury3   18m

NAME                                        READY   SYNCED   EXTERNAL-NAME   AGE
key.keyvault.azure.upbound.io/test-vault1   True    True     test-vault1     17m
key.keyvault.azure.upbound.io/test-vault2   True    True     test-vault2     17m
key.keyvault.azure.upbound.io/test-vault3   True    True     test-vault3     17m
key.keyvault.azure.upbound.io/test-vault4   True    True     test-vault4     17m
key.keyvault.azure.upbound.io/test-vault5   True    True     test-vault5     17m

NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/test   True    True     test            18m

NAME                                                   READY   SYNCED   EXTERNAL-NAME                          AGE
accesspolicy.keyvault.azure.upbound.io/test-vault-sp   True    True     a721469d-bdb1-4b57-8916-cb0fbd7cbf7c   18m

Apparently it breaks during some special condition we need to figure out.

[ytsarev]

Something similar was reported in the underlying provider hashicorp/terraform-provider-azurerm#12299

[ytsarev]

Can be reproduced with recreation

 k delete -f reproduce.yaml
vault.keyvault.azure.upbound.io "test-vault-yury" deleted
key.keyvault.azure.upbound.io "test-vault1" deleted
key.keyvault.azure.upbound.io "test-vault2" deleted
key.keyvault.azure.upbound.io "test-vault3" deleted
key.keyvault.azure.upbound.io "test-vault4" deleted
key.keyvault.azure.upbound.io "test-vault5" deleted
resourcegroup.azure.upbound.io "test" deleted
accesspolicy.keyvault.azure.upbound.io "test-vault-sp" deleted

// wait for full resource deletion, double check with `k get managed`

k get -f reproduce.yaml
NAME                                              READY   SYNCED   EXTERNAL-NAME    AGE
vault.keyvault.azure.upbound.io/test-vault-yury   True    True     testvaultyury3   8m40s

NAME                                        READY   SYNCED   EXTERNAL-NAME   AGE
key.keyvault.azure.upbound.io/test-vault1   False   True     test-vault1     8m40s
key.keyvault.azure.upbound.io/test-vault2   False   True     test-vault2     8m40s
key.keyvault.azure.upbound.io/test-vault3   False   True     test-vault3     8m40s
key.keyvault.azure.upbound.io/test-vault4   False   True     test-vault4     8m40s
key.keyvault.azure.upbound.io/test-vault5   False   True     test-vault5     8m40s

NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/test   True    True     test            8m40s

NAME                                                   READY   SYNCED   EXTERNAL-NAME

k describe key.keyvault.azure.upbound.io/test-vault1

    Message:               apply failed: A resource with the ID "https://testvaultyury3.vault.azure.net/keys/test-vault1/769eddd3e6ed4223bb397015d8fe12aa" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_key_vault_key" for more information.:

similar happens before and after the proposed fix.

[ytsarev]

Okay, I understand what is happening. After key vault is getting deleted, the old keys are getting automatically recreated due to default soft deleteion ( also mentioned in https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_key )

In this scenario, after we apply the manifests after deletion, the keys need to be imported with external-name annotation because they already exist in a recreated, previously soft-deleted key vault.

The import fails with the current implementation of the provider - the current keyVaultURLIDConf function does not have a randomized version in externalName, thus no way to properly import the resource following https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault_key#import

I've amended the draft PR

Now using the external-name annotation containing unique version string like

 apiVersion: keyvault.azure.upbound.io/v1beta1
  kind: Key
  metadata:
    name: test-vault2
    labels:
      name: test-vault2
      stage: sandbox
    annotations:
      crossplane.io/external-name: 0b8d7e086ff148b2bbc7ddf2ad9eab98

it is possible to successfully import the Key:

key.keyvault.azure.upbound.io/test-vault2   True    True     0b8d7e086ff148b2bbc7ddf2ad9eab98   21m

[ytsarev]

@jribmartins so to summarize, with the current version of provider, the creation of key vault works, but import is not. It means that the reproducer you provided will succeed with the creation of the key:

k get -f reproduce.yaml
NAME                                         READY   SYNCED   EXTERNAL-NAME    AGE
vault.keyvault.azure.upbound.io/test-vault   True    True     yuryvaultcust1   17m

NAME                                        READY   SYNCED   EXTERNAL-NAME   AGE
key.keyvault.azure.upbound.io/test-vault    True    True     test-vault      2m51s

NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/test   True    True     test            17m

NAME                                                     READY   SYNCED   EXTERNAL-NAME                          AGE
accesspolicy.keyvault.azure.upbound.io/test-vault-sp     True    True     a721469d-bdb1-4b57-8916-cb0fbd7cbf7c   17m

I just had to add - SetRotationPolicy to AccessPolicy to avoid access limitation failure - the creation works and you can use it right away for testing purposes.

But the external-name handling is not reliable, and the import is broken, so in case of e.g. provider pod restart, you can hit a situation similar to what @jeremypng describes.

The #536 should fully solve it - I will polish it early next week, and it will require some additional cycles on extended regression testing given the change substantial blast radius.

[swyngaard]

I'm seeing the same import issue when using Azure secrets. It seems your PR #536 will also resolve importing of secrets @ytsarev?

[ytsarev]

@swyngaard yes, that's the plan :) We are testing the vault Secrets as well before the merge of #536 👍