Provider failing to use TFC as a backend

[nalshamaajc]

provider-terraform/internal/terraform/terraform.go

Line 414 in b21417f

// Resources returns a list of resources in the Terraform state.

The issue started with me getting an error when supplying a TFC workspace name that is different than the one in the external-name annotation different changed the value of the external-name annotation to match the workspace I created and the "workspace not supported" error mentioned in this comment was gone.

I was now left with a different error which was complaining about the absence of a state file in this backend.

echo "H4sIAAAAAAAA/0yOMU7FQAxE+73Cb4YD8AXiHhRAl8psvFlLWTt4vYq4PUoUpF+48cy8mdv09treDT0oGEVWxk4dxYbOTyl9nu9GSgs31kC21kjnDuefIc6gh+gdH0MRVfq/L4mCMItzDvNf7JWd8cXuVMwbKnV8Myt8KMwxOiMq4/mCrrSkMGwmGqdwcRF2NG+cpUi+JqyWKcT0fpte2nHpDwAA//8BAAD//yMXozveAAAA" | base64 -d | gunzip
No state file was found!

State management commands require a state file. Run this command
in a directory where Terraform has been run or use the -state flag
to point the command to a specific state location.

What environment did it happen in?

• Crossplane Version: v1.14.0
• Provider Version: provider-terraform:v0.11.0
• Kubernetes Version: Client Version: v1.28.3 / Server Version: v1.25.15-eks-4f4795d
• Kubernetes Distribution: EKS

Expected Behavior

Use the workspace with no errors.

Code

cat <<EOF | kubectl replace --force -f -
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
    name: terraform
spec:
  credentials:
  - filename: .terraformrc # use exactly this filename by convention
    source: Secret
    secretRef:
      namespace: crossplane-system #upbound-system
      name: terraformrc
      key: .terraformrc
  - filename: .git-credentials # use exactly this filename
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: git-credentials
      key: .git-credentials
  configuration: |
    provider "aws" {
      region = "us-west-2"
    
      default_tags {
        tags = {
          Source             = "crossplane"
          Team               = "tag1"
          CostOrg            = "tag2"
          ProductLine        = "tag3"
          CrossplaneResource = "True"
          Env                = "dev"
        }
      }
    }
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      cloud {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXXXXTOKENXXXXXXX"
        workspaces {
          name = "crossplane-foundation"
        }
      }
    }
---
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  annotations:
    crossplane.io/external-name: "crossplane-foundation"
    meta.upbound.io/example-id: tf/v1beta1/workspace
  name: foundation-remote
spec:
  deletionPolicy: Delete 
  providerConfigRef:
    name: terraform
  forProvider:
    module: git::https://github.com/ORG/private-module?ref=branch
    source: Remote
    varFiles:
      - configMapKeyRef:
          key: crossplane.tfvars.json
          name: terraform-json
          namespace: default
        source: ConfigMapKey
        format: JSON
  writeConnectionSecretToRef:
    name: terraform-workspace
    namespace: default
EOF

Workaround

The workaround for this problem is seeding the state in the terraform cloud workspace with a data resource.

Code

Create a local file ex: providers.tf

    provider "aws" {
      region = "us-west-2"
    
      default_tags {
        tags = {
          Source             = "crossplane"
          Team               = "tag1"
          CostOrg            = "tag2"
          ProductLine        = "tag3"
          CrossplaneResource = "True"
          Env                = "dev"
        }
      }
    }
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      cloud {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXXTokenXXXXXXX"
        workspaces {
          name = "crossplane-foundation"
        }
      }
    }
    
data "aws_region" "current" {}

Output

Crossplane R&D % terraform init

Initializing Terraform Cloud...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 4.0"...
- Installing hashicorp/aws v4.67.0...
- Installed hashicorp/aws v4.67.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform Cloud has been successfully initialized!

You may now begin working with Terraform Cloud. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

Crossplane R&D % terraform apply
data.aws_region.current: Reading...
data.aws_region.current: Read complete after 0s [id=us-west-2]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Crossplane R&D % terraform state list
data.aws_region.current

[nalshamaajc]

FWIW using the same code without creating the Terraform cloud workspace causes the following behavior.

• The Workspace object creates a Terraform Cloud workspace.
• The Workspace fails to use the created Terraform Cloud workspace and returns an empty response.

Events:
  Type     Reason                         Age                 From                             Message
  ----     ------                         ----                ----                             -------
  Warning  CannotObserveExternalResource  73s (x3 over 3m6s)  managed/workspace.tf.upbound.io  cannot diff (i.e. plan) Terraform configuration: Terraform encountered an error. Summary: . To see the full error run: echo "H4sIAAAAAAAA/wAAAP//AQAA//8AAAAAAAAAAA==" | base64 -d | gunzip

It seems like it is waiting for a response the same way we are requested to select a Terraform workspace for our run.

[bobh66]

Does the content of the workspaces attribute in the cloud section need to match the value of the workspace name that is used by the provider? The provider user the external-name as the workspace name, so should that same name be included in the cloud workspaces list?

[nalshamaajc]

Well besides the exact name there are other options that can be used in the workspaces block like prefix and tags, but so far this only name works. If I have the tf-provider workspace create the Terraform Cloud workspace it creates it with no tags. Which means I can only reference the Terraform Workspace in the Cloud block using prefix which didn't work and name.

workspaces is required in the cloud block.
What are you suggesting @bobh66 ?

[kuisathaverat]

I have the same error with the elastic terraform provider, so I made a simple test to check it, and I found that a really basic example has the same issue. In this case, creating a data resource does no help, the terraform plan is completely valid and works on terraform generating a valid terraform state file.

---
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default-tf
spec:
  # This optional configuration block can be used to inject HCL into any
  # workspace that uses this provider config, for example to setup Terraform
  # providers.
  configuration: |
    terraform {
      required_version = ">= 1.5.0"
    }
---
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: example-inline
  annotations:
    # The terraform workspace will be named 'hello-world'. If you omitted this
    # annotation it would be derived from metadata.name - i.e. 'example-inline'.
    crossplane.io/external-name: example-inline
spec:
  forProvider:
    # For simple cases you can use an inline source to specify the content of
    # main.tf as opaque, inline HCL.
    source: Inline
    module: |
      resource "random_id" "example" {
        byte_length = 4
      }

      output "hello_world" {
        value = "Hello, World! - ${terraform.workspace}-${random_id.example.hex}"
      }

      resource "local_file" "example" {
        content  = "${random_id.example.hex}"
        filename = "${path.module}/example.txt"
      }

      data "template_file" "init" {
        template = ""
        vars = {
          consul_address = ""
        }
      }
  writeConnectionSecretToRef:
    namespace: default
    name: terraform-workspace-hello-world
  providerConfigRef:
    name: default-tf

I have tested with v0.10.0, v0.11.0 and v0.12.0

Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-12-01T13:02:56Z
    Message:               observe failed: cannot list Terraform resources: Terraform encountered an error. Summary: . To see the full error run: echo "H4sIAAAAAAAA/0yOMU7FQAxE+73Cb4YD8AXiHhRAl8psvFlLWTt4vYq4PUoUpF+48cy8mdv09treDT0oGEVWxk4dxYbOTyl9nu9GSgs31kC21kjnDuefIc6gh+gdH0MRVfq/L4mCMItzDvNf7JWd8cXuVMwbKnV8Myt8KMwxOiMq4/mCrrSkMGwmGqdwcRF2NG+cpUi+JqyWKcT0fpte2nHpDwAA//8BAAD//yMXozveAAAA" | base64 -d | gunzip

No state file was found!

State management commands require a state file. Run this command
in a directory where Terraform has been run or use the -state flag
to point the command to a specific state location.

[nalshamaajc]

@kuisathaverat your providerConfig isn't configuring Terraform Cloud as a backend so your code is probably using the default backend in this case.

  configuration: |
    terraform {
      required_version = ">= 1.5.0"
    }

[kuisathaverat]

@nalshamaajc 🤦‍♂️ I trusted the provider to manage the terraform state for me in the k8s cluster, but it does not do it. You MUST configure a backend, something I did not see in the documentation set as required. Also, the error does not help here to realize that the provider does not manage the default backend.

---
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default-tf
spec:
  configuration: |
    terraform {
      required_version = ">= 1.5.0"
      backend "kubernetes" {
        secret_suffix     = "k8s-backend"
        namespace         = "crossplane-system"
        in_cluster_config = true
      }
    }

[mydoomfr]

The terraform state configuration is documented in the provider's documentation in the marketplace : https://marketplace.upbound.io/providers/upbound/provider-terraform/v0.12.0/docs/quickstart

  configuration: |
    provider "google" {
      credentials = "gcp-credentials.json"
      project     = "YOUR-GCP-PROJECT-ID"
    }

    // Modules _must_ use remote state. The provider does not persist state.
    terraform {
      backend "kubernetes" {
        secret_suffix     = "providerconfig-default"
        namespace         = "upbound-system"
        in_cluster_config = true
      }
    }

However, I agree that this is not the default behavior we should expect if we omit it.

[bobh66]

https://github.com/upbound/provider-terraform#known-limitations

[nalshamaajc]

@nalshamaajc 🤦‍♂️ I trusted the provider to manage the terraform state for me in the k8s cluster, but it does not do it. You MUST configure a backend, something I did not see in the documentation set as required. Also, the error does not help here to realize that the provider does not manage the default backend.

---
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default-tf
spec:
  configuration: |
    terraform {
      required_version = ">= 1.5.0"
      backend "kubernetes" {
        secret_suffix     = "k8s-backend"
        namespace         = "crossplane-system"
        in_cluster_config = true
      }
    }

The docs mention somewhere that it will use whatever you configure as a backend if I'm not mistaken.

@kuisathaverat Did the workaround work for you?

[kuisathaverat]

The docs mention somewhere that it will use whatever you configure as a backend if I'm not mistaken.

yep, there is a comment in the terraform plans in the Quickstart and in the known limitations

@kuisathaverat Did the workaround work for you?

yes, using the k8s backend works like a charm, I have configured the Elastic Terraform provider without issues