Operator needs finalizer permission in RBAC to work in OpenShift

[aelbarkani]

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

We have an OKD cluster installed and we would like to use Config connector, which doesn't seem to work, probably because a problem in RBAC. It seems rights on finalizer subresources are needed

We have these error messages inside the operator pod:
forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on

Additional Diagnostic Information

Kubernetes Cluster Version

Client Version: v1.20.1-5-g76a04fc
Server Version: 4.6.0-0.okd-2021-02-14-205305
Kubernetes Version: v1.19.2-1049+f173eb4a83e557

Config Connector Version

1.42.0

Config Connector Mode

namespaced

Log Output

Error log:

{"severity":"info","msg":"applying manifest"}
{"severity":"info","msg":"executing kubectl","command":"kubectl","args":["apply","--validate=false","--force","-f","-"]}
{"severity":"error","msg":"error from running kubectl apply","stdout":"","stderr":"Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nError from server (Forbidden): error when creating \"STDIN\": customresourcedefinitions.apiextensions.k8s.io \"accesscontextmanageraccesslevels.accesscontextmanager.cnrm.cloud.google.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nError from server (Forbidden): error when creating \"STDIN\": customresourcedefinitions.apiextensions.k8s.io \"accesscontextmanageraccesspolicies.accesscontextmanager.cnrm.cloud.google.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nError from server (Forbidden): error when creating \"STDIN\": customresourcedefinitions.apiextensions.k8s.io \"accesscontextmanagerserviceperimeters.accesscontextmanager.cnrm.cloud.google.com\" i...
{"severity":"info","msg":"manifest:\n{\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"cnrm.cloud.google.com/version\":\"1.42.0\"},\"creationTimestamp\":null,\"labels\":{\"cnrm.cloud.google.com/managed-by-kcc\":\"true\",\"cnrm.cloud.google.com/system\":\"true\",\"cnrm.cloud.google.com/tf2crd\":\"true\",\"core.cnrm.cloud.google.com/configconnector\":\"configconnector.core.cnrm.cloud.google.com\"},\"name\":\"accesscontextmanageraccesslevels.accesscontextmanager.cnrm.cloud.google.com\",\"ownerReferences\":[{\"apiVersion\":\"core.cnrm.cloud.google.com/v1beta1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"ConfigConnector\",\"name\":\"configconnector.core.cnrm.cloud.google.com\",\"uid\":\"094bc3ee-1865-4888-916c-e4382ce7171c\"}]},\"spec\":{\"additionalPrinterColumns\":[{\"JSONPath\":\".metadata.creationTimestamp\",\"name\":\"Age\",\"type\":\"date\"},{\"JSONPath\":\".status.conditions[?(@.type=='Ready')].status\",\"description\":...
{"severity":"error","msg":"applying manifest","error":"error from running kubectl apply: exit status 1"}
{"severity":"error","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"configconnector-controller","request":"/configconnector.core.cnrm.cloud.google.com","error":"error applying manifest: error from running kubectl apply: exit status 1"}
{"severity":"info","logger":"NameChecker","msg":"preflight check before reconciling ConfigConnector","name":"configconnector.core.cnrm.cloud.google.com"

Steps to Reproduce

Steps to reproduce the issue

We have followed the procedure explained here:
https://cloud.google.com/config-connector/docs/how-to/install-other-kubernetes

[caieo]

Hi @aelbarkani, we're sorry you ran into this issue. I believe this may likely be due to a permissions issue specific to your cluster since we test on vanilla K8s clusters to ensure Config Connector can install properly. If possible, we recommend using GKE clusters to ensure Config Connector works.

After a brief investigation, I noticed some other GitHub issues mentioning a similar finalizer RBAC issue using OpenShift clusters and that OpenShift requires an additional RBAC permission for <resource>/finalizer to be able to set finalizers. We can look into adding additional finalizer permissions to our cluster roles. I've added this to our backlog and will update this thread when the working permissions are added.

[caieo]

@aelbarkani, unfortunately, due to the nature of this issue (only appearing in ConfigConnector installations on OpenShift clusters), we don't have a reliable way to test our changes.

One workaround you can try:

1. create your own custom ClusterRole binding with the appropriate finalizers for each of the resources that need the RBAC permissions
2. bind it to the operator's service account (which is the configconnector-operator service account in the configconnector-operator-system namespace)

We're really sorry we don't currently support your use-case, and (again) would recommend using GKE clusters for an ensured smooth experience with ConfigConnector.

[kaovilai]

Apart from my #797 PR, the only other change I had to do to make operator run on OpenShift is to remove runAsUser and runAsGroup from manager container's securityContext.

On OpenShift we enforce UID ranges for each namespace by default (which means the 1000 UID used by default won't work on OpenShift without messing with SecurityContextConstraints)

The patch to run additional to current config-connector installation docs is following

kubectl patch statefulset.apps configconnector-operator \
  --namespace configconnector-operator-system \
  --type json \
  --patch '[{"op": "remove", "path": "/spec/template/spec/containers/0/securityContext/runAsUser"}, {"op": "remove", "path": "/spec/template/spec/containers/0/securityContext/runAsGroup"}, {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities/drop", "value": ["ALL"]}]'

Where the last op is to remove a warning.

[kaovilai]

Looks like in addition to kaovilai@a756ff1

Other pods are going to need similar uid changes.. so perhaps I need to fork this operator.

[kaovilai]

Investigating creating SecurityContextConstraints to enable running the operator and its resources without maintaining a fork image.

[kaovilai]

Here's my wiki for getting ConfigConnector running on OpenShift.
https://github.com/kaovilai/k8s-config-connector/wiki/Using-Config-Connector-on-OpenShift

[diviner524]

@kaovilai Thank you for providing the wiki and creating the PR! We appreciate your efforts! Apologies for the delayed response, we will try to review your contribution sooner.

In the meantime, I wanted to check in on your investigation progress. Do you anticipate needing to fork the operator config to make it compatible with OpenShift?

[kaovilai]

The PR is applicable to technically not just openshift, as it is a k8s native admission controller that just may not be on by default.

I don't think we would wanna maintain a separate fork. The change is not much, mostly UID and perhaps creating SCC resource.

I would be happy to contribute to upstream an "openshift mode". Otherwise the most I'd do is just wiki.

I can write wiki here or add to readme here?

[kaovilai]

TL;DR: Issues on OpenShift with current ConfigConnector

• Not enough role permission for finalizer
  • can be fixed with that PR, applies to other k8s.
• runAsUser: 1000 is disallowed but on everything operator deploys
  • can be overridden by creating SecurityContextContraints
    • Could be documentation/wiki/script I contribute to this repo
  • create OpenShift mode (if you guys are willing)
  • fork (not ideal)
• hostNetworks, hostPorts usage
  • needs to create SecurityContextConstraints to allow workload to run.

The wiki resolves all three issues in a way that do not require changes in GCP upstream.

[diviner524]

@kaovilai Thanks for your contribution!
We have merged the PR. It will be included in our next release.

Also could you please start another PR to add the OpenShift install instructions?

Although we haven't yet finalized the document structure for this repository, you can start by adding a document under the path "docs/openshift". We can reorganize the folders as needed in the future.