Short term solution for "OutOfcpu" issues for runners

[BeyondEvil]

We have an issue where runners get OutOfcpu even tho their QoS is guaranteed.

This seems to be a known issue and we're waiting on AWS to release a new EKS AMI with the fix.

Here's a (short term) solution to deal with that problem.

https://gist.github.com/BeyondEvil/456ef05f8b31f693e1018abf6b998fd0

[Diontsoumas]

@BeyondEvil out of curiosity, you have both requests and limits in your runners?

[BeyondEvil]

@Diontsoumas

Correct. Both for memory and CPU.

    resources:
      limits:
        cpu: 1500m
        memory: 4Gi
      requests:
        cpu: 1500m
        memory: 4Gi

qosClass: Guaranteed

[Diontsoumas]

We are currently testing internally the actions controller and I was wondering which is the best way to figure out those resources. I can easily image a scenario where we keep adding more memory because the test suit throws OOM errors.

[BeyondEvil]

We are currently testing internally the actions controller and I was wondering which is the best way to figure out those resources. I can easily image a scenario where we keep adding more memory because the test suit throws OOM errors.

That depends on what you're going to run in the workflows.