Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JWTOIDCAuthEngineConfig does not Reconcile correctly after creating CR without OIDCCredentials #225

Open
dabelenda opened this issue Jan 23, 2024 · 13 comments

Comments

@dabelenda
Copy link

Hello,

As a followup for #165 and the fix #167 there are two (tightly coupled) issues I found.

When creating a CR of Kind JWTOIDCAuthEngineConfig without OIDCCredentials, for example:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  name: gitlab-oidc-config
  namespace: vault-config-operator
spec:
  path: gitlab
  boundIssuer: 'https://gitlab.example.com'
  OIDCDiscoveryURL: 'https://gitlab.example.com'

The vault-config-operator creates the equivalent configuration in vault, but injects default values in the CR, which results in CR being:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  name: gitlab-oidc-config
  namespace: vault-config-operator
spec:
  path: gitlab
  boundIssuer: 'https://gitlab.example.com'
  OIDCDiscoveryURL: 'https://gitlab.example.com'
  OIDCCredentials:
    passwordKey: password
    usernameKey: username
  JWKSURL: ''
  OIDCResponseMode: ''
  OIDCClientID: ''
  namespaceInState: true
  defaultRole: ''

In the new state of the CR, any modification will perform unwanted changes in vault, and when trying to remove the OIDCCredentials field the following errors are written in the vault-config-operator:

2024-01-22T14:30:57Z DEBUG reconcile {"controller": "jwtoidcauthengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "JWTOIDCAuthEngineConfig", "JWTOIDCAuthEngineConfig": {"name":"gitlab-oidc-config","namespace":"vault-config-operator"}, "namespace": "vault-config-operator", "name": "gitlab-oidc-config", "reconcileID": "4c94d7b6-57d2-4892-abd4-c119ea6207c4", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "JWTOIDCAuthEngineConfig", "namespace": "vault-config-operator", "name": "gitlab-oidc-config"}}
2024-01-22T14:30:57Z ERROR unable to prepare internal values {"controller": "jwtoidcauthengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "JWTOIDCAuthEngineConfig", "JWTOIDCAuthEngineConfig": {"name":"gitlab-oidc-config","namespace":"vault-config-operator"}, "namespace": "vault-config-operator", "name": "gitlab-oidc-config", "reconcileID": "4c94d7b6-57d2-4892-abd4-c119ea6207c4", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "JWTOIDCAuthEngineConfig", "namespace": "vault-config-operator", "name": "gitlab-oidc-config"}, "error": "no means of retrieving a secret was specified"}
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic
/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:95
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65
github.com/redhat-cop/vault-config-operator/controllers.(*JWTOIDCAuthEngineConfigReconciler).Reconcile
/home/runner/work/vault-config-operator/vault-config-operator/controllers/jwtoidcauthengineconfig_controller.go:80
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
2024-01-22T14:30:57Z ERROR unable to complete reconcile logic {"controller": "jwtoidcauthengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "JWTOIDCAuthEngineConfig", "JWTOIDCAuthEngineConfig": {"name":"gitlab-oidc-config","namespace":"vault-config-operator"}, "namespace": "vault-config-operator", "name": "gitlab-oidc-config", "reconcileID": "4c94d7b6-57d2-4892-abd4-c119ea6207c4", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "JWTOIDCAuthEngineConfig", "namespace": "vault-config-operator", "name": "gitlab-oidc-config"}, "error": "no means of retrieving a secret was specified"}
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:67
github.com/redhat-cop/vault-config-operator/controllers.(*JWTOIDCAuthEngineConfigReconciler).Reconcile
/home/runner/work/vault-config-operator/vault-config-operator/controllers/jwtoidcauthengineconfig_controller.go:80
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
2024-01-22T14:30:57Z DEBUG events no means of retrieving a secret was specified {"type": "Warning", "object": {"kind":"JWTOIDCAuthEngineConfig","namespace":"vault-config-operator","name":"gitlab-oidc-config","uid":"b412ea66-5990-4589-b301-d0e3dfef3205","apiVersion":"redhatcop.redhat.io/v1alpha1","resourceVersion":"88217988"}, "reason": "ProcessingError"}
2024-01-22T14:30:57Z ERROR Reconciler error {"controller": "jwtoidcauthengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "JWTOIDCAuthEngineConfig", "JWTOIDCAuthEngineConfig": {"name":"gitlab-oidc-config","namespace":"vault-config-operator"}, "namespace": "vault-config-operator", "name": "gitlab-oidc-config", "reconcileID": "4c94d7b6-57d2-4892-abd4-c119ea6207c4", "error": "no means of retrieving a secret was specified"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
@trevorbox
Copy link
Contributor

Try using the latest version of the operator (you may need to update the CRDs). That OIDCCredentials field should be optional.

Here are the CR defaults from my test...

apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  name: gitlab-oidc-config
  namespace: test-vault-config-operator
spec:
  JWKSCAPEM: ""
  JWKSURL: ""
  OIDCClientID: ""
  OIDCDiscoveryCAPEM: ""
  OIDCDiscoveryURL: https://gitlab.example.com
  OIDCResponseMode: ""
  boundIssuer: https://gitlab.example.com
  defaultRole: ""
  namespaceInState: true
  path: gitlab

@trevorbox trevorbox mentioned this issue Feb 2, 2024
@dabelenda
Copy link
Author

Hello @trevorbox,

I am using 0.8.25 installed by OLM on OCP4, so if this is not the latest version I am not sure on how to fix it.

But what I report, is that when creating the CR without OIDCCredentials it is then present when you "read after write". The applied configuration applied on Vault does not contain the clientid client-secret parts, so I find it very weird.
The CR fields do not represent the actual state that is being sync`ed and modifying the CR after creation leads to problems.

@trevorbox
Copy link
Contributor

I cannot replicate the issue so perhaps you need to update the CRD for that api (it may have changed between upgrades of the operator from olm). Can you compare the CRD on your cluster vs https://github.com/redhat-cop/vault-config-operator/blob/main/config/crd/bases/redhatcop.redhat.io_jwtoidcauthengineconfigs.yaml#L71-L162 . Notice that there is no default.

@dabelenda
Copy link
Author

https://github.com/redhat-cop/vault-config-operator/blob/main/config/crd/bases/redhatcop.redhat.io_jwtoidcauthengineconfigs.yaml#L77 and https://github.com/redhat-cop/vault-config-operator/blob/main/config/crd/bases/redhatcop.redhat.io_jwtoidcauthengineconfigs.yaml#L133 are exactly what I find inside my final object.

I realized something, I use ArgoCD to manage the objects, so maybe it reads the CRD, finds these default values and adds them by itself?

@erlisb
Copy link
Contributor

erlisb commented May 24, 2024

Hi @dabelenda, is this still an issue ? Thnx

@dabelenda
Copy link
Author

I still have something weird (using 0.8.27 now).

When telling ArgoCD to create the CR without the field OIDCCredentials, the resulting manifest contains it with value:

  OIDCCredentials:
    passwordKey: password
    usernameKey: username

Which are the defaults described in the CRD, so probably if the fields are optional they should not contain defaults so that tools like ArgoCD avoid injecting them.

Then what happens from there is even more weird, the first reconcile works, everything is "fine", the operator did not try to use OIDCCredentials secret (which is not defined !), and ArgoCD thinks the field is set as intended.

But if I edit the CR to modify it (in particular if I remove the field OIDCCredentials), then the reconcile fails with:

    - lastTransitionTime: '2024-05-27T06:29:26Z'
      message: no means of retrieving a secret was specified
      observedGeneration: 2
      reason: LastReconcileCycleFailed
      status: 'False'
      type: ReconcileFailed

I see inconsistencies that leads to this issue:

  • the absence of fields secret, randomSecret and vaultSecret is considered an error by the operator (maybe only in certain cases during reconcile).
  • The whole OIDCCredentials is optional but subfields like passwordKey and usernameKey have default values so tools that follow the CRD definition might add them as specified.

@erlisb
Copy link
Contributor

erlisb commented Jun 1, 2024

Hi @dabelenda, can you please try a plain Vault + VCO Installation (without ArgoCD) and let us know ?
We would like to know how we can move on with this issue.

@dabelenda
Copy link
Author

Testing without ArgoCD is not easy with how the cluster is created and managed.

I don't have direct network access to the k8s API.

Does it mean that this interaction between ArgoCD and VCO is not a use-case you consider supported?

@erlisb
Copy link
Contributor

erlisb commented Jun 4, 2024

Hi,
I am just asking you to do a plain Vault+VCO installation, locally. Using kind/minikube and both charts of Vault and VCO.
This is how I am testing if the provided CRD works as expected, as this shouldn't be related with ArgoCD, afaik.

@erlisb
Copy link
Contributor

erlisb commented Oct 31, 2024

Hi @dabelenda, is this still an issue ?
Did you tried to install/test VCO as I suggested?

Thank you

@raffaelespazzoli I don't see an issue here currently.

@dabelenda
Copy link
Author

Hello @erlisb,

Sorry for the delay, I had issues with minikube and not a lot of time to troubleshoot it.

I finally managed to get something conclusive:

$ minikube-linux-amd64 config set rootless true
[...]
$ minikube-linux-amd64 start --driver=podman --container-runtime=containerd
[...]
$ helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true --create-namespace
[...]
$ helm install vault-config-operator vault-config-operator/vault-config-operator -n vault --set enableMonitoring=false --set enableCertManager=true --create-namespace
[...]
$ helm ls -n vault
NAME                  NAMESPACE REVISION  UPDATED                                 STATUS    CHART                         APP VERSION
vault-config-operator vault     1         2024-11-01 07:51:09.731495585 +0100 CET deployed  vault-config-operator-v0.8.29 v0.8.29
$ cat test.yml
---
apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  name: gitlab-oidc-config-2
  namespace: vault
spec:
  path: gitlab
  boundIssuer: 'http://gitlab.example.com'
  OIDCDiscoveryURL: 'http://gitlab.example.com'
$ kubectl create -f test.yml --dry-run=server -o yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  creationTimestamp: "2024-11-01T06:52:01Z"
  generation: 1
  name: gitlab-oidc-config-2
  namespace: vault
  uid: d1efee54-9ce5-40d5-abb4-8b60f4834638
spec:
  JWKSCAPEM: ""
  JWKSURL: ""
  OIDCClientID: ""
  OIDCCredentials:
    passwordKey: password
    usernameKey: username
  OIDCDiscoveryCAPEM: ""
  OIDCDiscoveryURL: http://gitlab.example.com
  OIDCResponseMode: ""
  authentication:
    path: kubernetes
    serviceAccount:
      name: default
  boundIssuer: http://gitlab.example.com
  defaultRole: ""
  namespaceInState: true
  path: gitlab

This seems to confirm my initial guess that something puts the default values for the OIDC client authentication.

@dabelenda
Copy link
Author

To continue the investigation:

$ kubectl apply -f test.yml  --server-side
jwtoidcauthengineconfig.redhatcop.redhat.io/gitlab-oidc-config-2 serverside-applied
$ kubectl get jwtoidcauthengineconfig.redhatcop.redhat.io/gitlab-oidc-config-2 -n vault -o yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  creationTimestamp: "2024-11-01T08:31:35Z"
  generation: 1
  name: gitlab-oidc-config-2
  namespace: vault
  resourceVersion: "9546"
  uid: d2391d6c-638a-4d44-84ae-14b0cab36b9c
spec:
  JWKSCAPEM: ""
  JWKSURL: ""
  OIDCClientID: ""
  OIDCCredentials:
    passwordKey: password
    usernameKey: username
  OIDCDiscoveryCAPEM: ""
  OIDCDiscoveryURL: http://gitlab.example.com
  OIDCResponseMode: ""
  authentication:
    path: kubernetes
    serviceAccount:
      name: default
  boundIssuer: http://gitlab.example.com
  defaultRole: ""
  namespaceInState: true
  path: gitlab
status:
  conditions:
  - lastTransitionTime: "2024-11-01T08:32:11Z"
    message: 'Put "https://127.0.0.1:8200/v1/auth/kubernetes/login": dial tcp 127.0.0.1:8200:
      connect: connection refused'
    observedGeneration: 1
    reason: LastReconcileCycleFailed
    status: "False"
    type: ReconcileFailed

This is the case even without the --server-side option:

$ kubectl delete jwtoidcauthengineconfig.redhatcop.redhat.io/gitlab-oidc-config-2 -n vault
$ kubectl apply -f test.yml
jwtoidcauthengineconfig.redhatcop.redhat.io/gitlab-oidc-config-2 created
$ kubectl get jwtoidcauthengineconfig.redhatcop.redhat.io/gitlab-oidc-config-2 -n vault -o yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: JWTOIDCAuthEngineConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"redhatcop.redhat.io/v1alpha1","kind":"JWTOIDCAuthEngineConfig","metadata":{"annotations":{},"name":"gitlab-oidc-config-2","namespace":"vault"},"spec":{"OIDCDiscoveryURL":"http://gitlab.example.com","boundIssuer":"http://gitlab.example.com","path":"gitlab"}}
  creationTimestamp: "2024-11-01T08:34:39Z"
  generation: 1
  name: gitlab-oidc-config-2
  namespace: vault
  resourceVersion: "9772"
  uid: df0b1827-4657-45f3-a402-b32486a3e1b9
spec:
  JWKSCAPEM: ""
  JWKSURL: ""
  OIDCClientID: ""
  OIDCCredentials:
    passwordKey: password
    usernameKey: username
  OIDCDiscoveryCAPEM: ""
  OIDCDiscoveryURL: http://gitlab.example.com
  OIDCResponseMode: ""
  authentication:
    path: kubernetes
    serviceAccount:
      name: default
  boundIssuer: http://gitlab.example.com
  defaultRole: ""
  namespaceInState: true
  path: gitlab
kubectl version                                                      
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

@erlisb
Copy link
Contributor

erlisb commented Nov 9, 2024

@dabelenda sorry, but I still don't get it what issue are you experiencing.

Can you elaborate a bit better?

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants