dex secret not found

[donydonald1]

hello @khuedoan
i am a big fan of this and I have been trying to get work like this a week now but I have a little issue trying to make mine work. hoping you could help.
external secret didn't create a secret for dex.
please help

)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dex
    Optional:    false
  kube-api-access-wl57w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason  Age                    From     Message
  ----     ------  ----                   ----     -------
  Warning  Failed  60m (x12 over 62m)     kubelet  Error: secret "dex-secrets" not found
  Normal   Pulled  2m31s (x278 over 62m)  kubelet  Container image "ghcr.io/dexidp/dex:v2.38.0" already present on machine ``` 

[khuedoan]

Hi, dex-secrets is created by https://github.com/khuedoan/homelab/blob/master/platform/dex/templates/secret.yaml, could you please post the output of:

kubectl describe -n dex externalsecret dex-secrets

[donydonald1]

+ kubectl describe -n dex externalsecret dex-secrets
Name:         dex-secrets
Namespace:    dex
Labels:       argocd.argoproj.io/instance=dex
Annotations:  <none>
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  Creation Timestamp:  2024-04-18T09:48:36Z
  Generation:          1
  Resource Version:    51172
  UID:                 94eb9cd1-310b-4a3d-8574-7ed4b326de5c
Spec:
  Data:
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_id
    Secret Key:             KANIDM_CLIENT_ID
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             KANIDM_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.grafana
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GRAFANA_SSO_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.gitea
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GITEA_CLIENT_SECRET
  Refresh Interval:         1h
  Secret Store Ref:
    Kind:  ClusterSecretStore
    Name:  global-secrets
  Target:
    Creation Policy:  Owner
    Deletion Policy:  Retain
    Name:             dex-secrets
Status:
  Conditions:
    Last Transition Time:  2024-04-18T09:48:36Z
    Message:               could not get secret data from provider
    Reason:                SecretSyncedError
    Status:                False
    Type:                  Ready
Events:
  Type     Reason        Age                    From              Message
  ----     ------        ----                   ----              -------
  Warning  UpdateFailed  4m40s (x24 over 109m)  external-secrets  error retrieving secret at .data[0], key: kanidm.dex, err: secrets "kanidm.dex" not found 

this is also affecting other deployment as well and for some reasons none of the secrets generated works when trying to login to the deployments

woodpecker          pre-install-agent-secret-check-jsqrs                     0/1     Completed                    0                75m
woodpecker          woodpecker-agent-5b6945cc7b-8c49l                        0/1     CrashLoopBackOff             19 (2m41s ago)   75m
woodpecker          woodpecker-agent-5b6945cc7b-nrmmf                        0/1     CrashLoopBackOff             19 (2m52s ago)   75m

[kikokikok]

Same problem for me, I think the
kanidm.dex key is never creqted in the global-secrets ClusterSecretStore

[khuedoan]

kandim.dex should be created by default in the post install script, could you try running make post-install manually?

[kikokikok]

Well the postscript fails when calling the reset of users with the python k8s client. It doesn't return the expected json payload on the stdout as expected which causes an error on json deserialization.
When executing with a remote ssh into the container, I see the json paylod

bash-5.2# make postinstall
make: *** No rule to make target 'postinstall'.  Stop.
bash-5.2# make post-install
Traceback (most recent call last):
  File "/home/cklat/homelab/./scripts/hacks", line 256, in <module>
    main()
  File "/home/cklat/homelab/./scripts/hacks", line 247, in main
    kanidm_login(["admin", "idm_admin"])
  File "/home/cklat/homelab/./scripts/hacks", line 158, in kanidm_login
    password = reset_kanidm_account_password(account)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cklat/homelab/./scripts/hacks", line 152, in reset_kanidm_account_password
    return json.loads(resp)['password']
           ^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 2 (char 1)

Manual bash inside the container:

kanidmd recover-account --output json admin
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: This is running as uid == 0 (root) which may be a security risk.
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: permissions on /data/server.toml may not be secure. Should be readonly to running uid. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml owned by the current uid, which may allow file permission changes. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: DB folder /data has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 INFO     ｉ [info]: Running account recovery ...
{"password":"VU29tSLcAqjccXWez12dQKhKNuPNWcJDcQ34NXK1gGGFSGwN"}

[numpythuckles]

Was running into the same JSON decode error on the reset_kanidm_account_password function.

Ran ./scripts/hacks outside of nix was able to create global-secrets for:

gitea.renovate
gitea.woodpecker

Renovate was able to create its own renovate-secret afterwards and began submitting PRs to Git.

An Oauth2 application was created in Gitea for Woodpecker also.

Still not seeing any kanidm.* in global-secrets. Working on it.

[eenikad]

@numpythuckles are you still working on this issue? Ran into the same problem and would appreciate every possible hint.

[brimdor]

getting this issue as well, any progress on it?

[brimdor]

Temporarily created this yaml that I kubectl apply when post-install fails in order to manually create the kanidm.dex
I place this in the scripts folder and called it kanidm_fix.yaml then ran kubectl apply -f scripts/kanidm_fix.yaml assuming this is being run from the root homelab folder.

The issue still persists though. The post-install errors every time. This temp fix only creates the secret but it does not apply the attributes in the other commands for the function that is failing.

apiVersion: batch/v1
kind: Job
metadata:
  name: secret-generator-kanidm-fix
  namespace: global-secrets
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: secret-generator
          image: lachlanevenson/k8s-kubectl:latest
          command:
            - sh
            - -c
          args:
            - |
              apk add --no-cache openssl
              CLIENT_ID=$(openssl rand -base64 32)
              CLIENT_SECRET=$(openssl rand -base64 64)
              kubectl create secret generic kanidm.dex \
                --from-literal=client_id="${CLIENT_ID}" \
                --from-literal=client_secret="${CLIENT_SECRET}" \
                -n global-secrets
      serviceAccount: secret-generator

[lfleal]

Attempting to apply the above results in the following error:
error: resource mapping not found for name: "secret-generator-kanidm-fix" namespace: "global-secrets" from "scripts/kanidm_fix.yaml": no matches for kind "Job" in version "" ensure CRDs are installed first

[brimdor]

That's strange. I built and ran this after post-install failed. Did you run it at that point in the process?

[lfleal]

First time I ran it:
error: error validating "scripts/kanidm_fix.yaml": error validating data: apiVersion not set; if you choose to ignore these errors, turn validation off with --validate=false
Running with --validate=false generated the error above. The file is missing apiVersion, does that exist in your file?

[brimdor]

Ah ok, it was there but it was next to the code block indicators so it cut it off. Its fixed.

[lfleal]

Adding apiVersion: batch/v1 to the beginning of the yaml above fixes the error.

[brimdor]

Yep, as I stated, it is fixed.

[lfleal]

After getting the secrets added, I'm getting this now:
error parse config file /tmp/dex.config.yaml-461377176: error unmarshaling JSON: parse connector config: invalid character '\n' in string literal

[khuedoan]

Thanks for reporting the issues everyone, I think the ./scripts/hacks have grown big and messy, so I'll rewrite it to make it a bit more reliable

[khuedoan]

The following commits should fix the issue in the short term:

• eb1381d
• 209a950