How add annotations to ARC listeners?

[adamchalat]

In latest v0.5.0 version was added option to monitor behavior of runner scale set.
https://github.com/actions/actions-runner-controller/tree/master/docs/gha-runner-scale-set-controller/samples/grafana-dashboard

How can I add required annotations to listener pods using provided Helm charts?
https://github.com/actions/actions-runner-controller/tree/master/charts

Currently no annotations are inherited.
I don't see such possibility in this PR: #2758

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-09-15T09:26:28Z"
  labels:
    actions.github.com/organization: HASH
    actions.github.com/scale-set-name: general-standard
    actions.github.com/scale-set-namespace: scale-set
    app.kubernetes.io/component: runner-scale-set-listener
    app.kubernetes.io/part-of: gha-runner-scale-set
    app.kubernetes.io/version: 0.5.0
    runner-spec-hash: HASH
  name: general-standard-HASH-listener
  namespace: scale-set

[Link-]

You can do that in https://github.com/actions/actions-runner-controller/releases/tag/gha-runner-scale-set-0.6.0 which came out today.

[adamchalat]

I was trying to do that, by adding annotations, but this is not working for me as expected.
Maybe, I'm doing something wrong?

  listenerTemplate:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: /metrics

[Link-]

@nikola-jokic any thoughts here? We might be overriding those if metrics are enabled

[nikola-jokic]

I think this is related to the nil map bug in annotations that is fixed in this PR.

[wburnett]

@Link- / @nikola-jokic I'm still seeing this on version 0.8.1 of the chart, I'd like to reopen this discussion - also happy to open a new thread if that's more appropriate.

@adamchalat did you ever get this working?

I'm using the listenerTemplate field as provided in the gha-runner-scale-set example values file: https://github.com/actions/actions-runner-controller/blob/master/charts/gha-runner-scale-set/values.yaml#L93

That section of my values file is as follows:

listenerTemplate:
  metadata:
    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: true

The pod does not appear to receive any of these annotations:

 kubectl describe pod -n arc-systems arc-runner-set-golang-754b578d-listener                                                                                                            
Name:             arc-runner-set-golang-754b578d-listener
Namespace:        arc-systems
Labels:           actions.github.com/organization=<snip>
                  actions.github.com/scale-set-name=arc-runner-set-golang
                  actions.github.com/scale-set-namespace=arc-runners
                  app.kubernetes.io/component=runner-scale-set-listener
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.8.1
                  runner-spec-hash=7d4df5874
Annotations:      <none>

I also did some digging in the golang and found the place where the listener gets reconciled: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/autoscalinglistener_controller.go#L75

I dug back through the types and found that the field does appear to exist under: autoscalingListener.Spec.Template.ObjectMeta.Annotations, however the annotations field never appears to be set anywhere in the reconcile function, or any of its helpers that I could find.

It looks like it's intended to be set here, but digging back, it never actually gets surfaced to the template values: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/resourcebuilder.go#L289

mergeListenerWithPodTemplate gets called here during newScaleSetListenerPod: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/resourcebuilder.go#L282

And newScaleSetListener is called here during createListenerPod: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/autoscalinglistener_controller.go#L480

createListenerPod is called here during Reconcile: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/autoscalinglistener_controller.go#L242

Going through that chain, it looks like if autoscalingListener.Spec.Template were set, mergeListenerPodWithTemplate would add the annotations, but spec.template never gets set anywhere that I can find.

As a quick and dirty test to my theory, I cloned the repo locally and changed the test here: https://github.com/actions/actions-runner-controller/blob/master/controllers/actions.github.com/resourcebuilder_test.go#L28

I added a "something": else" key to the annotations map[string]string on 28, and then on line 61 added the following assertion:
assert.Equal(t, "else", listener.Annotations["something"]) which came back with a failure.

Please let me know if there's any other relevant info I can provide, and I'd also be happy to start working on a PR to fix the issue if you think I'm looking in the right direction.

[krohmag]

The crux of @wburnett's issue is knowing how to add an annotation to the pod that an AutoScalingListener resource produces. Based on the provided example, it looks like they're trying to add a safe eviction annotation to the listener pod(s) to allow the cluster autoscaler to reschedule the pods on other nodes.

The newAutoScalingListener function does not appear to have any logic to easily disseminate annotations to the pods produced by the AutoScalingListener resource via the template.

Additionally, this got me looking at the tests for the ephemeral runner set, where I found something else that seems off. The annotation in the tests is the only annotation that actually shows up in the annotations for the resource. Adding a t.Log(ephemeralRunnerSet.Annotations) returns a map with only the tested key/value. Digging into the newEphemeralRunnerSet function, it appears that there's supposed to be logic to add new annotations here, but there's no inclusion of any other annotations other than the AnnotationKeyGitHubRunnerGroupName key and its value.

Definitely seems like there's a gap here.

[nikola-jokic]

Hey @wburnett,

I am failing to reproduce the issue where annotations are not applied. Basically, the annotation used in the listenerTemplate is only applied to the listener pod and not the AutoscalingListener resource.
What I did is apply values.yaml file with the following modification:

listenerTemplate:
  metadata:
    annotations:
      example: test
      cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
  spec:
    containers:
      - name: listener

This one produced a listener pod:

❯ k get po -n arc-systems main-scaleset-754b578d-listener -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    example: test

Could you please write a step-by-step guide how to reproduce the issue?

[krohmag]

@nikola-jokic - If you're going to use the listenerTemplate field, do you have to include a complete template spec?

[nikola-jokic]

@krohmag you don't need to provide the entire spec. The way it works is if the field is set by us, and the template field you provided is not empty, we will override our default value. Otherwise, we will just apply the value from the spec.

@wburnett are you talking about the annotation on the AutoscalingListener resource, or on the actual listener pod? I did try to upgrade as well, and it worked. If you are inspecting the autoscalinglistener resource, then we are not applying annotations to it. We are only providing the annotation to the actual listener pod.

[wburnett]

@nikola-jokic no, this was the listener pod, not the AutoscalingListener resource. Re-ran the command and included the describe:

kubectl describe pod -n arc-systems arc-runner-set-golang-754b578d-listener                                                                                                      
Name:             arc-runner-set-golang-754b578d-listener
Namespace:        arc-systems
Priority:         0
Service Account:  <redacted>
Node:             <redacted>
Start Time:       Tue, 23 Jan 2024 16:34:10 -0500
Labels:           actions.github.com/organization=myfintech
                  actions.github.com/scale-set-name=arc-runner-set-golang
                  actions.github.com/scale-set-namespace=arc-runners
                  app.kubernetes.io/component=runner-scale-set-listener
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.8.1
                  runner-spec-hash=c9c5cd79
Annotations:      <none>