Issues when creating runners in vcluster

[shankarRaman]

Controller Version

0.25.1

Helm Chart Version

0.20.1

CertManager Version

1.1.0

Deployment Method

Helm

cert-manager installation

Yes cert-manager installed through proper procedure

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: krogertechnology-runner-deployment
spec:
  template:
    spec:
      image: docker-prod.registry.kroger.com/tools/actions-runner-dind:v2.292.0-ubuntu-20.04-no-proxy
      imagePullPolicy: IfNotPresent
      organization: krogertechnology
      labels:
      - vcluster-dev
      - aks-dev
      group: AKS Non-prod
      dockerdWithinRunnerContainer: true
      dockerRegistryMirror: https://docker.registry.kroger.com/
      resources:
        limits:
          cpu: "8"
          memory: "8Gi"
        requests:
          cpu: "1"
          memory: "8Gi"
      env:
      - name: DISABLE_RUNNER_UPDATE
        value: "true"

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: krogertechnology-runner-deployment-autoscaler
spec:
  scaleTargetRef:
    name: krogertechnology-runner-deployment
  minReplicas: 1
  maxReplicas: 1
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.5'    # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
    scaleDownThreshold: '0.2'   # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
    scaleUpAdjustment: 10      # The scale up runner count added to desired count
    scaleDownAdjustment: 1    # The scale down runner count subtracted from the desired count

To Reproduce

1. Install the ARC using Helm
2. Create the runner deployments using the above manifest

Observe error

Describe the bug

We get this error
Error from server (InternalError): error when creating "aks/runner-deployment-aks.yaml": Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": failed to call webhook: Post "https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=10s": proxy error from 127.0.0.1:6443 while dialing 192.168.0.215:9443, code 500: 500 Internal Server Error

Describe the expected behavior

We expect the runner pods to be created.

Controller Logs

https://gist.github.com/shankarRaman/37954ad16b7725e2769b819576f7d0d7#file-shankar-arc-log

Runner Pod Logs

N/A

Additional Context

This is a vcluster setup using k3s

[mumoshu]

@shankarRaman Hey! If you don't mind, can we move this to Discussions so that everyone interested in vcluster can participate?
We don't officially support vcluster yet and it should just work on any standard K8s cluster, which basically means this issue is vcluste-specific that ARC has nothing to do with.
Your controller logs says ARC is working fine without any errors.
Perhaps in a vcluster setup there's a middle-man proxy between vcluster-managed K8s control-plane and the ARC's controller-manager that is hosted directly within the host K8s cluster?

[shankarRaman]

@mumoshu thanks for starting this discussion. That's what I want to also understand, how do I determine if there's a proxy configured?

[mumoshu]

@shankarRaman Umm.. I have no idea. Perhaps you can review some vcluster architecture documentation and see how K8s control-plane is supposed to communicate with pods and services in the host cluster? 🤔