How can we install 2 controllers in same cluster with different namespace using Helm?

[roshvin]

Checks

• I've already read https://github.com/actions-runner-controller/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.

Controller Version

0.25.2

Helm Chart Version

0.25.2

CertManager Version

0.20.2

Deployment Method

Helm

cert-manager installation

yes

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

helm install

To Reproduce

Release "github-actions-runner" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "github-actions-runner-actions-runner-controller-proxy" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "actions-runner-system_fdc": current value is "actions-runner-system"

Describe the bug

Release "github-actions-runner" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "github-actions-runner-actions-runner-controller-proxy" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "actions-runner-system_fdc": current value is "actions-runner-system"

Describe the expected behavior

how we can install 2 controller in same cluster with different namespace.

Controller Logs

Release "github-actions-runner" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "github-actions-runner-actions-runner-controller-proxy" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "actions-runner-system_fdc": current value is "actions-runner-system"

Runner Pod Logs

Release "github-actions-runner" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "github-actions-runner-actions-runner-controller-proxy" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "actions-runner-system_fdc": current value is "actions-runner-system"

Additional Context

No response

[mumoshu]

how we can install 2 controller in same cluster with different namespace.

Clearly, this isn't a bug, but a question!

[mumoshu]

Check singleNamespace and relevant helm values. Do differentiate release name because it affects non-namespaced k8s resource names.

[roshvin]

Hi @mumoshu thanks . i installed in 2 different name space

and when i give below value in helm. for diff name space . the runner is not registering .

scope:
  # If true, the controller will only watch custom resources in a single namespace
  singleNamespace: true
  # If `scope.singleNamespace=true`, the controller will only watch custom resources in this namespace
  # The default value is "", which means the namespace of the controller
  watchNamespace: "my-namespace-1"

when i make singleNamespace: false - runer is registring .

am i missing something . could you pls suggest

[mumoshu]

Did you differentiate release names? Sharing the full reproduction steps and the whole ARC logs would always help reduce this kind of back-and-forth and make quick investigation possible! Thanks

[dongho-jung]

I'm doing exactly the same, installing 2 ARC in the same cluster with different namespaces using Helm.

I think I found a way. the key points were fullnameOverride and scope. refer to the below

fullnameOverride: actions-runner-controller-dev
scope:
   singleNamespace: true
...

[roshvin]

Hi @0xF4D3C0D3 thanks for the hit. now both controller started . but in the second controller runner is not registering .

could you pls suggest if i am missing anything.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: dot-free-runner-poc-fdc
spec:
  githubAPICredentialsFrom:
        secretRef:
          name: controller-manager
  ephemeral: true
  enterprise: xxxx
  dockerEnabled: false
  dockerdWithinRunnerContainer: false
  labels:
      - dot-enterprise-runner
      - dependabot
  # Other mandatory fields from StatefulSet
  selector:
    matchLabels:
      app: dot-free-runner-poc
  serviceName: dot-free-runner-poc
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/runner: unconfined
      labels:
        app: dot-free-runner-poc
    spec:

     securityContext:
          runAsUser: 1000
          fsGroup: 1000

[dongho-jung]

how did you install that? and how did you get those logs (what did you type for getting those logs)? I installed ARC with helm like helm upgrade --install --namespace actions-runner-system-dev --create-namespace --wait actions-runner-controller actions-runner-controller/actions-runner-controller -f values.yml or helm upgrade --install --namespace actions-runner-system --create-namespace --wait actions-runner-controller actions-runner-controller/actions-runner-controller -f values.yml

and the difference between two value.yamls is like below

I hope this can help you

[roshvin]

HI @0xF4D3C0D3 thanks for the quick update .

we also use same helm command to install . below are the diff in 2 values.yaml

helm upgrade -i github-actions-runner-fdc --namespace actions-runner-system-fdc  --create-namespace --version 0.20.2 -f runner-values_25.2.yaml_FDC actions-runner-controller/actions-runner-controller --recreate-pods

helm upgrade -i github-actions-runner --namespace actions-runner-system  --create-namespace --version 0.20.2 -f runner-values_25.2.yaml actions-runner-controller/actions-runner-controller --recreate-pods

 diff runner-values_25.2.yaml runner-values_25.2.yaml_FDC
1c1
< fullnameOverride: actions-runner-system
---
> fullnameOverride: actions-runner-system-fdc

63c63
<         - path: /github-runner
---
>         - path: /github-runnerfdc

[mumoshu]

Thanks! And you're still setting an error like the below one, right?

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "github-actions-runner-actions-runner-controller-proxy" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "actions-runner-system_fdc": current value is "actions-runner-system"

Then I think you've a dangling helm release in another namespace. I can think of no other reason that can result in duplicated clusterroles with same name(can you?).

To double-check, you could see the helm chart template for the metadata.name of the clusterrole resource. As long as it contains {{ .Release.Name }} in it, it shouldn't have resulted in the duplicated clusterroles.

[dongho-jung]

I had the same error indeed and I solved it with setting fullnameOverride

[roshvin]

Hi @mumoshu now i am not seeing the resource exists error. only problem is in the second namespace controller the runners are not getting registered

[dongho-jung]

If you post the output that makes you think the runners are not getting registered then it's better to get help

[mumoshu]

Also- Do you have by any chance use a custom runner image? Can it be the cause of your problem @roshvin ?

[roshvin]

Hi , pls find the runner register log

i am using same image / same runnerset yaml in fist controller and different namespace and its working .

[mumoshu]

If you run kubectl get po -o yaml $RUNNER_POD on the problematic pod, do you see anything missing in the pod spec? At least RUNNER_TOKEN env needs to be present. Taking diff between good and bad pod manifests with diff --unified good-pod.yaml bad-pod.yaml would help.

[roshvin]

hi @mumoshu thanks for the hint. RUNNER_TOKEN is not percent in non-working runner pod.

i have created token like this kubectl create secret generic controller-manager --from-literal=github_token=xxxxxxxxxxx

and reffering it in RS . am i missing something . pls sugget.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: dot-free-runner-poc-fdc
spec:
  githubAPICredentialsFrom:
        secretRef:
          name: controller-manager
  ephemeral: true
  enterprise: xxxxx
  dockerEnabled: false
  dockerdWithinRunnerContainer: false
  labels:
      - dot-enterprise-runner
      - dependabot
  # Other mandatory fields from StatefulSet
  selector:
    matchLabels:
      app: dot-free-runner-poc
  serviceName: dot-free-runner-poc
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/runner: unconfined
      labels:
        app: dot-free-runner-poc
    spec:

     securityContext:
          runAsUser: 1000
          fsGroup: 1000