Multiple runner deployments with multiple horizontal autoscalars

[pankaj-verma-by]

Hello

I am using ARC (App version - 0.24.0) and have deployed ARC in an AKS cluster. My organization has a restriction to create "org" level private runners so I am using repository level runners to create private runners in multiple github repositories. I am using RunnerDeployment(s) backed with HorizontalRunnerAutoscaler(s). Following is my RunnerDeployment and its corresponding HPA definition for 1 repo. The same set (RD + HPA) is created for other repos also:

---

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: abc-runner
namespace: actions-runner-system
spec:
template:
spec:
nodeSelector:
decisioning: "true"
repository: org1/abc
env: []
labels:

• private-runners
  resources:
  requests:
  cpu: "11"
  memory: "40Gi"

---

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: abc-runner-deployment-autoscaler
spec:
scaleDownDelaySecondsAfterScaleOut: 120
scaleTargetRef:
name: abc-runner
metrics:

• type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  repositoryNames:
  • org1/abc
    minReplicas: 1
    maxReplicas: 3

---

The above kind of k8s resource definition goes for other repositories like this:

---

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: xyz-runner
namespace: actions-runner-system
spec:
template:
spec:
nodeSelector:
decisioning: "true"
repository: org1/xyz
env: []
labels:

• private-runners
  resources:
  requests:
  cpu: "11"
  memory: "40Gi"

---

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: xyz-runner-deployment-autoscaler
spec:
scaleDownDelaySecondsAfterScaleOut: 120
scaleTargetRef:
name: xyz-runner
metrics:

• type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  repositoryNames:
  • org1/xyz
    minReplicas: 1
    maxReplicas: 3

---

I am only using pull based autoscaling and not webhook server one due to some DNS restrictions at org level.

Issue:

The issue is that autoscaling works fine only for 1 repo which means when I run multiple workflow jobs for "abc" repo then I can see three runner pods being deployed and get scaled out after 10 mins (not in 2 minutes - another issue) but when I execute multiple workflow jobs from "xyz" repo then I can see only the default 1 runner pod for "xyz" repo. The other 2 pods do not spin up and remain in the queue until first job gets completed. Once first job gets completed then also no additional pod comes up. The job gets executed on default runner pod only (Due to "minReplicas: 1")

Please do help to figure out the issue here. Thanks

[pankaj-verma-by]

Please provide some help in fixing this issue. Thanks

[mumoshu]

Hey! Probably you might be able to investigate it further by looking for ARC logs that contain Suggested desired replicas of https://github.com/actions-runner-controller/actions-runner-controller/blob/8161136cbd7451e055cbe4ca5cf9729ff1fe141e/controllers/autoscaling.go#L215

If you configured everything as you described, it should just work. As it doesn't, there might be any misconfigurations or unseen bugs underneath.

[pankaj-verma-by]

This message appears though there are multiple workflow runs are still in queue. There were 2 workflows in queue and 1 running. It seems ARC created only 1 pod and other 2 were terminated:

2022-07-14T12:40:26ZDEBUGcontroller-runtime.webhook.webhooksreceived request{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "b2edd3f0-8fcd-4fc3-
98aa-db54005ca1fc", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-07-14T12:40:26ZINFOrunner-resourcevalidate resource to be created{"name": "availability-runner-rpkn7-z4w82"}
2022-07-14T12:40:26ZDEBUGcontroller-runtime.webhook.webhookswrote response{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "
b2edd3f0-8fcd-4fc3-98aa-db54005ca1fc", "allowed": true}
2022-07-14T12:40:26ZDEBUGcontroller-runtime.webhook.webhooksreceived request{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "d14c4333-a4af-4097-
bfdf-54251b124290", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-07-14T12:40:26ZINFOrunner-resourcevalidate resource to be updated{"name": "availability-runner-rpkn7-b2kwf"}
2022-07-14T12:40:26ZDEBUGcontroller-runtime.webhook.webhookswrote response{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "
d14c4333-a4af-4097-bfdf-54251b124290", "allowed": true}
2022-07-14T12:40:26ZINFOactions-runner-controller.runnerRemoved finalizer{"runner": "actions-runner-system/availability-runner-rpkn7-b2kwf"}
2022-07-14T12:40:26ZDEBUGactions-runner-controller.runnerreplicasetCreated replica(s){"runnerreplicaset": "actions-runner-system/availability-runner-rpkn7", "lastSyncTi
me": null, "effectiveTime": "", "templateHashDesired": "64884875d9", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserve
d": [], "created": 1}
2022-07-14T12:40:26ZDEBUGactions-runner-controller.runnerreplicasetSkipped reconcilation because owner is not synced yet{"runnerreplicaset": "actions-runner-system/availab
ility-runner-rpkn7", "owner": "actions-runner-system/availability-runner-rpkn7-z4w82", "pods": null}

[mumoshu]

Hey. You'd better share a more complete ARC logs. The Created replica(s) message only says it added a runner for whatever reason. What we might need to know is how, when, and why ARC thought it was enough to just add one replica, instead of two or three.