do not store tailscale_tailnet_key's sensitive information in terraform state.

[james-lawrence]

Is your feature request related to a problem? Please describe.
terraform isn't a secret storage system. sensitive information should ideally not be stored in it. this is related to #144 and tailscale/tailscale#3243. I understand this might not be feasible (depends on the backend for the apis involved among other things). just thought I'd point this out.

Describe the solution you'd like
ideally a way to fetch the sensitive information at runtime and only store a checksum in the terraform state for verifying the value.

[DentonGentry]

There are several Hashicorp vault providers which can handle tailscale key state:
https://github.com/bloominlabs/vault-plugin-secrets-tailscale
https://github.com/davidsbond/vault-plugin-tailscale

The most recent updates in tailscale/tailscale#3243 describe the intended solution: an OAuth 2.0 Client Credential flow with a long-lived credential. That credential can be used to issue short lifetime access tokens which would presumably not be stored anywhere.

[dgivens]

I tried using vault-plugin-secrets-tailscale, but it didn't work out for me. When a Vault lease is revoked or expires, the secrets associated with the lease are revoked.

I am using Nomad to run Terraform. When it runs a job, the job is given a Vault token that is revoked upon completion of the job. In the instance of Terraform being run as a batch job, when I used the Tailscale Vault plugin to generate a preauth key that would be inserted into the instance userdata, it ended up being revoked before cloud-init could be run.

The only way I found to work around it was to switch to this provider.

[james-lawrence]

@DentonGentry the existence of other solution doesn't negate the problems with this one.

[DentonGentry]

@DentonGentry the existence of other solution doesn't negate the problems with this one.

If not pulling secrets from Hashicorp vault or similar solutions, then what would you propose?

[davidsbond]

ideally a way to fetch the sensitive information at runtime and only store a checksum in the terraform state for verifying the value.

I think part of the problem with this solution is that the Tailscale API only returns the secret key data once on creation and it cannot be retrieved again. So that secret value has to go in the state for now. You're welcome to try out the Vault plugin that I implemented https://github.com/davidsbond/vault-plugin-tailscale which is admittedly very lightweight, but I'd be happy to handle contributions or feature requests on it.

[james-lawrence]

If not pulling secrets from Hashicorp vault or similar solutions, then what would you propose?

I proposed a solution to the problem already in the issue. as I said I understand technical underpinning of tailscales credential management make solving this problematic atm. but the terraform integration is problematic as it opens up sensitive values to be exposed inadvertently. alternative options don't absolve this terraform resource from being problematic in that regard is all I was asserting.

I mainly opened this so this problem is discoverable for others.

[dgivens]

To be fair, this is a known issue with Terraform. It's very explicitly stated in the Terraform docs about avoiding secrets in the state. It's not something specific to the tailscale provider. There are mitigation steps you can take, such as restricting access to the location where states are stored, ensuring the states are kept in a location that encrypts the data at rest, etc.

For my implementation, I decided to keep the Tailscale API creds in a simple kv Vault secret, which the Vault Terraform provider reads, then uses to configure the Tailscale provider. The Tailscale keys created by Terraform are single use and ephemeral, so they aren't really of any use to anyone who might get access to the state once they've been used.

provider "vault" {}

data "vault_kv_secret_v2" "tailscale" {
  mount = "secret"
  name  = "tailscale"
}

locals {
  tailscale_data = jsondecode(data.vault_kv_secret_v2.tailscale.data_json)
}

provider "tailscale" {
  api_key = local.tailscale_data.api_key
  tailnet = local.tailscale_data.tailnet
}

resource "tailscale_tailnet_key" "foo" {
  reusable      = false
  ephemeral     = true
  preauthorized = true
  tags = [
    "tag:foo"
  ]
}

[james-lawrence]

@dgivens it very much is specific to this provider if this provider is doing the exact thing terraform says not to do. ;)

[davidsbond]

The terraform SDK does have a helper for modifying the value before it enters the state:

https://github.com/hashicorp/terraform-plugin-sdk/blob/3495894b26ae61cf11da2669170ff9d1c655896b/helper/schema/schema.go#L204

However, I think it would cause issues for a reusable key on a subsequent apply. Unless you can keep obtaining the secret value from the API I'm not sure what else the provider can do to keep reusable keys working.

[dgivens]

@james-lawrence I can see your point of view, but it applies equally to the Vault provider as well. The first several paragraphs of the overview for that provider is very clear about the risk of secrets in state and plan files, including this highlighted block:

Important

Interacting with Vault from Terraform causes any secrets that you read and write to be persisted in both Terraform's state file and in any generated plan files. For any Terraform module that reads or writes Vault secrets, these files should be treated as sensitive and protected accordingly.

[james-lawrence]

@dgivens yes i don't use the vault provider for that reason. not sure what you're trying to get at here. if one person jumps off a bridge should you?

[kevcube]

@james-lawrence you're not addressing the fact that terraform is known to store secrets in state. Almost every provider has this "issue" with certain resources.

This provider is not particularly engineered to store these secrets, but they are an attribute of a resource - something that is always stored in terraform state. This is a requirement for them to be consumed by any other terraform code.

The only way to avoid this is for terraform to never set the value tailscale_tailnet_key.key. Which makes it useless, as it can only be read once.

Your solution of "fetch the sensitive information at runtime" from where? You're welcome to use terraform to write the key into vault or wherever else, but ultimately terraform needs to be able to read it from tailscale_tailnet_key.

Please re-read the official HashiCorp view on secrets in state and treat your state as if it is confidential. https://developer.hashicorp.com/terraform/language/state/sensitive-data

I don't see what you're describing as an issue with this provider.

[james-lawrence]

@kevcube you don't need to store sensitive information to validate sensitive data. you can store a checksum instead of the sensitive data; as I pointed out in the top level issue. this issue is to act as a warning to people using the provider and to maybe prod the developers to stop leaking secrets when it isn't necessary.

[kevcube]

@james-lawrence this package is not a validator, it is a provider. It provides the key after it is created. This is not at all a fault of this packages' developers, and this is not secrets leakage. Honestly: have you used terraform before this? And also, have you read the documentation I linked regarding secrets in terraform state?

[james-lawrence]

@kevcube its a resource; it doesn't need to store the secret to check if remote state has changed. Yes I also understand that given how terraform works and how currently tailscales api is design its storing the secret in the state file because it has no where else to store it.

You did bother to read my original statement right in the first part of the issue correct?

I understand this might not be feasible (depends on the backend for the apis involved among other things). just thought I'd point this out.

I'm not making claims about the usefulness of this for those who don't care or treat terraform state as sensitive.
I'm not demanding it be fixed.

have you read the documentation I linked regarding secrets in terraform state

did you? terraform doesn't recommend you store sensitive data in the state. it tells you that if you do then you need to treat it as sensitive and gives you suggestions for doing so. you do realize that every backend for retrieving the state file exposes the sensitive data as plaintext? thus this particular resource exposes the entirety of tailnet to whoever grabs that file. given VPNs are often used as a network boundary thats a pretty big deal.

[sedlund]

terraform doesn't recommend you store sensitive data in the state.

Hashicorp's documentation says that sensitive data in the state file is 'inevitable' (: incapable of being avoided or evaded).

They recommend that you store it on a remote state provider, as terraform only keeps the state locally in memory, not written to disk, and is versioned and encrypted remotely.

This design choice is documented upstream with terraform and solutions are provided for a secure configuration. If you need local state store, OpenTofu implemented local state encryption.

[james-lawrence]

encryption doesn't do anything to resolve the problem. as I pointed out earlier anyone with access to terraform can read the data with a simple terraform state show/get .... (encryption at rest doesn't resolve that, nor does remote state storage) even if they don't have credentials to manage tailscale.

there are better technical mechanisms (like storing checksums and key stretching) that can be used to not store sensitive data in a system that isn't designed for security storage.

hashicorp is incorrect that its 'inevitable'. if tailscale decides to improve their terraform integration and their apis around management of a tailscale network they could make it vastly more user friendly and more secure. but that's up to them.

[sedlund]

anyone with access to terraform can read the data with a simple terraform state show/get

If one needs to partition access there are numerous methods to implement that, such as using HCP Terraform, or using a gitops model with something like Atlantis, where you can define role based permissions and approvals. Both run terraform remotely an end user never has access to the CLI or the state file.

Implementing either of these options is a far more secure design than hoping someone doesn't implement a provider in your infrastructure code that inadvertently leaks something deemed sensitive, and the current process lets anyone access the terraform cli.

[james-lawrence]

no its really not. it requires additional effort and maintenance by everyone vs fixing the core problem tailscale has with its provider. for everyone else in the future: no one is saying you cant setup terraform in a manner that would be reasonably secure. its just a lot of work, maintenance, and additional costs.

tailscale is particularly unique in the sense that vpns are already used as a security boundary. having such highly sensitive credentials stored in a system from which data can be easily extracted from is problematic (as demonstrated by the additional steps one needs to take to even approach a reasonably secure setup. especially when there are fairly trivial ways to not have the problem in the first place). unfortunately that requires tailscale to implement key stretching + salting within their credentials api and checksums in their terraform provided resources.

saying you can do N additional steps and costs to make it somewhat secure completely misses the problem/point of this issue; which is secure by design vs secure by ad hoc policies/implementations.

also note: atlantis/hcp terraform don't actual solve the issue because I as a developer can get terraform to emit basically any data I want to a file and then read it back as output via terraform configs exposing it which will then show up in the PR. wonderbar now credentials are exposed and stored in yet another system. essentially any system where the output of terraform is displayed doesn't solve this problem.