panic: tsnet: mkdir /var/lib/loft/tailscale: permission denied on k8s

[lucasfcnunes]

Logs

{"level":"info","ts":1727902209.6826887,"caller":"initialize/context.go:64","msg":"Initialize...","component":"loft"}
{"level":"info","ts":1727902209.682772,"caller":"initialize/context.go:119","msg":"Initialize agent...","component":"loft"}
{"level":"info","ts":1727902209.6827931,"caller":"initialize/context.go:142","msg":"Ensure agent crds...","component":"loft"}
{"level":"info","ts":1727902209.6827905,"caller":"initialize/context.go:74","msg":"Ensure certificates...","component":"loft"}
{"level":"info","ts":1727902209.6828368,"caller":"initialize/context.go:87","msg":"Ensure crds...","component":"loft"}
{"level":"info","ts":1727902209.6828673,"caller":"initialize/context.go:129","msg":"Ensure agent certificates...","component":"loft"}
{"level":"info","ts":1727902210.3038318,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-ingress-wakeup-agent-server-cert"}
{"level":"info","ts":1727902210.3038561,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-webhook-agent-cert"}
{"level":"info","ts":1727902210.3038738,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-apiservice-agent-cert"}
{"level":"info","ts":1727902210.312556,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-apiservice-agent-cert"}
{"level":"info","ts":1727902210.312903,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-ingress-wakeup-agent-server-cert"}
{"level":"info","ts":1727902210.3129025,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-webhook-agent-cert"}
{"level":"info","ts":1727902210.509248,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-cert"}
{"level":"info","ts":1727902210.509292,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-server-cert"}
{"level":"info","ts":1727902210.509324,"caller":"secretstore/secretstore.go:45","msg":"Ensure certificate secret","component":"loft","namespace":"vcluster-platform","secret":"loft-api-service-cert"}
{"level":"info","ts":1727902210.5133374,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-cert"}
{"level":"info","ts":1727902210.513338,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-api-service-cert"}
{"level":"info","ts":1727902210.5134916,"caller":"secretstore/secretstore.go:68","msg":"Found valid certificate secret","component":"loft","secret":"vcluster-platform/loft-server-cert"}
{"level":"info","ts":1727902212.077267,"caller":"crdbuilder/crdbuilder.go:156","msg":"Upgrade crd schema","component":"loft","crdName":"virtualclusterinstances.storage.loft.sh"}
{"level":"info","ts":1727902212.077774,"caller":"crdbuilder/crdbuilder.go:156","msg":"Upgrade crd schema","component":"loft","crdName":"spaceinstances.storage.loft.sh"}
{"level":"info","ts":1727902212.0784512,"caller":"crdbuilder/crdbuilder.go:156","msg":"Upgrade crd schema","component":"loft","crdName":"devpodenvironmenttemplates.storage.loft.sh"}
{"level":"info","ts":1727902212.0788016,"caller":"crdbuilder/crdbuilder.go:156","msg":"Upgrade crd schema","component":"loft","crdName":"devpodworkspaceinstances.storage.loft.sh"}
{"level":"info","ts":1727902214.6573207,"caller":"initialize/context.go:151","msg":"Done initializing agent...","component":"loft"}
{"level":"info","ts":1727902214.6573837,"caller":"initialize/context.go:104","msg":"Done initializing...","component":"loft"}
{"level":"info","ts":1727902214.667566,"logger":"controller-runtime.webhook","caller":"webhook/server.go:183","msg":"Registering webhook","component":"loft","path":"/quota"}
{"level":"info","ts":1727902214.6692815,"caller":"version/version.go:37","msg":"Starting with version","component":"loft","Kubernetes Version":"v1.30.3-gke.1969001","Platform Version":"4.1.0-alpha.16"}
{"level":"info","ts":1727902215.5061333,"caller":"initialize/manager.go:11","msg":"Starting k8s manager","component":"loft"}
{"level":"info","ts":1727902215.5062475,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server","component":"loft"}
{"level":"info","ts":1727902215.5063493,"logger":"controller-runtime.webhook","caller":"webhook/server.go:191","msg":"Starting webhook server","component":"loft"}
{"level":"info","ts":1727902215.5065017,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","component":"loft","bindAddress":"127.0.0.1:12000","secure":false}
{"level":"info","ts":1727902215.5067003,"logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:161","msg":"Updated current TLS certificate","component":"loft"}
{"level":"info","ts":1727902215.5068564,"logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","component":"loft","host":"","port":9443}
{"level":"info","ts":1727902215.5069833,"logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:115","msg":"Starting certificate watcher","component":"loft"}
{"level":"info","ts":1727902215.607936,"caller":"initialize/license.go:25","msg":"Starting license manager","component":"loft"}
{"level":"info","ts":1727902215.6079907,"caller":"initialize/configwatcher.go:29","msg":"Starting config watcher","component":"loft"}
{"level":"info","ts":1727902215.7482195,"caller":"initialize/configwatcher.go:213","msg":"Loft Router Domain configured","component":"loft","domain":"xhepxaf.loft.host"}
{"level":"error","ts":1727902215.748303,"caller":"initialize/configwatcher.go:216","msg":"Write loft router token","component":"loft","error":"write loft router domain: open /var/lib/loft/loft-domain.txt: permission denied"}
{"level":"info","ts":1727902216.4501948,"caller":"initialize/tailscale.go:27","msg":"Starting ts coordinator","component":"loft"}
{"level":"info","ts":1727902216.450273,"logger":"config-watcher","caller":"config/config.go:127","msg":"loft will load config from secret","component":"loft","defaultSecretName":"loft-manager-config"}
{"level":"info","ts":1727902216.5515184,"caller":"initialize/apiserver.go:66","msg":"Starting loft api server","component":"loft"}
{"level":"info","ts":1727902216.555346,"caller":"initialize/apigateway.go:28","msg":"Starting loft api gateway","component":"loft"}
{"level":"info","ts":1727902216.5555112,"caller":"oidc/authenticator.go:305","msg":"OIDC: No x509 certificates provided, will use host's root CA set","component":"loft"}
{"level":"info","ts":1727902216.555513,"caller":"runner/runner.go:164","msg":"Starting sleep mode runner","component":"loft"}
{"level":"info","ts":1727902216.5556624,"caller":"accesskeys/accesskey_runner.go:59","msg":"Starting access key runner","component":"loft"}
{"level":"info","ts":1727902216.5556667,"caller":"audit/audit.go:201","msg":"Audit enabled with policy","component":"loft","logFile":"/var/lib/loft/audit.log","usedAuditConfig":{"rules":[{"level":"None","resources":[{"group":"*","resources":["selfsubjectaccessreviews","subjectaccessreviews"]},{"group":"management.loft.sh","resources":["agentauditevents","licensetokens","directclusterendpointtokens","ingressauthtokens","selves","virtualclusterinstances/kubeconfig"]},{"group":"cluster.loft.sh","resources":["chartinfos"]}]},{"level":"Metadata","verbs":["create"],"resources":[{"group":"management.loft.sh","resources":["ownedaccesskeys","resetaccesskeys","sharedsecrets","clusters","users/profile","virtualclusterinstances/kubeconfig"]}],"omitStages":["RequestReceived"]},{"level":"Metadata","resources":[{"resources":["pods/log","pods/exec","pods/portforward"]},{"group":"management.loft.sh","resources":["tasks/log","virtualclusterinstances/log"]}],"omitStages":["RequestReceived"]},{"level":"None","verbs":["get","list","watch"]},{"level":"Request","verbs":["create"],"omitStages":["RequestReceived"]},{"level":"Metadata","omitStages":["RequestReceived"]}]}}
{"level":"info","ts":1727902216.555774,"caller":"datastoredriver/datastore.go:85","msg":"configured database connection pooling","component":"loft","maxIdleConns":2,"maxOpenConns":1,"connMaxLifetime":0}
{"level":"error","ts":1727902216.5561516,"caller":"datastore/backend.go:164","msg":"failed creating schema resources","component":"loft","error":"sqlite: check foreign_keys pragma: reading schema information unable to open database file: out of memory (14)"}
{"level":"error","ts":1727902216.556193,"caller":"handler/loft.go:69","msg":"Error initializing audit","component":"loft","error":"sqlite: check foreign_keys pragma: reading schema information unable to open database file: out of memory (14)"}
{"level":"info","ts":1727902216.5580711,"caller":"cache/shared_informer.go:313","msg":"Waiting for caches to sync for *generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]","component":"loft"}
{"level":"info","ts":1727902216.5593987,"caller":"admission/plugins.go:157","msg":"Loaded 1 mutating admission controller(s) successfully in the following order: MutatingAdmissionWebhook.","component":"loft"}
{"level":"info","ts":1727902216.5594277,"caller":"admission/plugins.go:160","msg":"Loaded 2 validating admission controller(s) successfully in the following order: ValidatingAdmissionPolicy,ValidatingAdmissionWebhook.","component":"loft"}
{"level":"info","ts":1727902216.6623795,"caller":"initialize/tailscale.go:14","msg":"Starting local control plane client","component":"loft"}
panic: tsnet: mkdir /var/lib/loft/tailscale: permission denied

goroutine 595 [running]:
github.com/loft-sh/loft/v4/pkg/tailscale/controlplane.(*ControlPlane).Start(0x7f94880, {0x5849470, 0xc0015b9560}, {0xc0116a40c0, 0x40})
    github.com/loft-sh/loft/v4/pkg/tailscale/controlplane/controlplane.go:106 +0x630
created by github.com/loft-sh/loft/v4/cmd/loft/initialize.startLocalControlPlaneClient in goroutine 1
    github.com/loft-sh/loft/v4/cmd/loft/initialize/tailscale.go:21 +0x16e
{"level":"error","ts":1727902216.7063143,"caller":"cmd/main.go:107","msg":"error executing root command","component":"loft","error":"setup restart: error running loft: exit status 2"}
Stream closed EOF for vcluster-platform/loft-5f667889dd-qf8tw (manager)

Versions

• Kubernetes v1.30.3-gke.1639000
• loft/loft 4.1.0-alpha.16 (image: ghcr.io/loft-sh/loft:4.1.0-alpha.16)

Workaround

Disable persistence

# ...
persistence:
  enabled: false

[deniseschannon]

Where are these logs from?

I see you referencing the loft version/image, so is this a vcluster specific issue or logs in loft?

[lucasfcnunes]

Logs in loft (in manager container) this chart https://artifacthub.io/packages/helm/loft/loft/4.1.0-alpha.16. I deployed it in a namespace called vcluster-platform.

[deniseschannon]

Can you share your full values.yaml?