Runner Pod can't start up with ServiceAccount on EKS 1.15.10 #16

missedone · 2020-03-21T02:55:29Z

I tried to deploy a runner with IAM Role for ServiceAccount (IRSA) in EKS 1.15.10, but the runner pod can't start up at all:

kubectl get pod
NAME                          READY   STATUS        RESTARTS   AGE
my-runner-lmzjc-znh8c         0/1     Terminating   0          10s

the manifest i used:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: actions-runner-system-sa
  namespace: actions-runner-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE>
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: my-runner
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      repository: example/repo
      serviceAccountName: actions-runner-system-sa
      automountServiceAccountToken: true
      containers:
        - name: runner
          image: example/action-runner:latest
          imagePullPolicy: Always

look into the event, i saw:

2s          Warning   FailedMount                pod/my-runner-zsvsd-vfdpz                Unable to mount volumes for pod "my-runner-zsvsd-vfdpz_actions-runner-system(1b78d4fb-25e6-4457-bd47-04ea06ae7d14)": timeout expired waiting for volumes to attach or mount for pod "actions-runner-system"/"my-runner-zsvsd-vfdpz". list of unmounted volumes=[aws-iam-token actions-runner-system-sa-token-8gmq9]. list of unattached volumes=[aws-iam-token actions-runner-system-sa-token-8gmq9]

The text was updated successfully, but these errors were encountered:

mumoshu · 2020-03-21T03:13:33Z

To clarify, containers[] is not the only option here. The workaround is to use spec.image to use a custom runner image so that serviceAccountName and automountServiceAccountToken works 😃

The fix for containers[] would be to add the required volume mounts to the runner container when containers[] is specified, which would look very similar to #14

missedone · 2020-03-21T04:00:01Z

it doesn't matter if use containers[] or image, same error.

mumoshu · 2020-03-21T04:43:18Z

Odd. Could you dump your runner pod by running kubectl get to -o yaml my-runner-lmzjc-znh8c?

mumoshu · 2020-03-21T04:49:28Z

Does it work if you tried to use the pod IAM role from a regular pod, not a pod managed by the actions-runner-controller?

alexandrst88 · 2020-03-23T09:48:02Z

I tested this on vanilla Kubernetes setup and service account was mounted to Runner Container without any issues.

nachomillangarcia · 2020-07-08T15:57:10Z

I think it's related to EKS IRSA. It works for me when I remove the annotation eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE>.

But with that annotation present, the controller keeps looping creating and destroying the runner pod.

EKS IRSA introduces envvars and volumes automatically in the pod:

  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true

env
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::xxxxxx:role/xxxxx
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Maybe those collide with some validation at the controller.

I've checked the pod definitions when using EKS IRSA and they're fine. It's the controller who destroys the pod.

callum-tait-pbx · 2020-08-26T14:09:05Z

@summerwind @mumoshu I'm running into the same problem here and have the same symptoms / experience as the nachomillangarcia. This prevents me using the solution as role based auth is a requirement and we only use EKS for our k8s needs. We are running EKS version 1.17 atm.

gregorygtseng · 2020-09-02T17:08:57Z

It's possible to specify the environment variables and mount the volumes manually in your Runner or RunnerDeployment spec. Not ideal but it works.

callum-tait-pbx · 2020-09-03T08:48:09Z

@gregorygtseng I've been able to do this however my runner is still using the underlying node role and failing to assume. If I apply the annotations magic sauce manually and then try to perform an assume role with the github action (see below) I get a permission denied User: arn:aws:sts::***:assumed-role/eksctl-eks-test-nodegroup-NodeInstanceRole-EKV6S90QD0F3/i-0c5ccd732adce9984 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::***:role/example-role

with:
    aws-region: us-west-1
    role-to-assume: arn:aws:iam::$AWS_ACCOUNT:role/example-role
    role-skip-session-tagging: true

Have you been able to assume into roles successfully?

gregorygtseng · 2020-09-03T17:00:02Z

Yes I am able to login to ECR and push our Docker images from a GitHub Actions workflow.
I'm not using the action you're using with role-to-assume however.

Does your user have the permission for sts:AssumeRole ?

Here's my RunnerDeployment if that helps. Note I had to add securityContext to allow reading of the token

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment
spec:
  replicas: 2
  template:
    spec:
      ...
      env:
        - name: AWS_ROLE_ARN
          value: arn:aws:iam::xxxxxx:role/xxxxx
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      serviceAccountName: actions-runner-sa
      automountServiceAccountToken: true
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: actions-runner-token-xxx
          readOnly: true
        - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
          name: aws-iam-token
          readOnly: true
      securityContext:
        fsGroup: 65534 # fix reading service token
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token

I would try a debug job where you sleep in the workflow and try execing into the runner?

$ aws sts get-caller-identity
{
    "UserId": "ABCDEFGHIJ:botocore-session-12345",
    "Account": "1234567890",
    "Arn": "arn:aws:sts::1234567890:assumed-role/example-role/botocore-session-12345"
}

callum-tait-pbx · 2020-09-03T17:30:38Z

Got it working, it was the magic combo of:

      - name: AWS_ROLE_ARN
        value: arn:aws:iam::$AWS_ACOUNT_ID:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      volumeMounts:
      - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
        name: aws-iam-token 
        readOnly: true
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      securityContext:
        fsGroup: 27

and the annotation on the service account. Used 27 as I couldn't sudo as runner if I used the nobody group for some reason.

gregorygtseng · 2020-09-03T18:01:04Z

That's great! Did you mean removing the annotation on the service account? Because that's what I had to do since we're doing the mounting manually to not interfere

callum-tait-pbx · 2020-09-04T09:34:46Z

No without the annotation I was not able to get IAM working with IAM. I needed to include:

The annotation to the base IAM role in my service account, without this auth wouldn't work
Add in envs, volumeMounts and volume that the annotation automatically injects. The apps manager seemed to kill the pod if you let the annotation do this, I pressume because it fails some sort of validation?
Add in the fsGroup so the token is readable, 65534 or 27 seemed to work. 65534 broke sudo though whereas 27 didn't

My final yaml (with aws account id's replaced with $AWS_ACCOUNT_ID)

kind: ServiceAccount
metadata:
  name: sre-actions-runner-system-sa-ajh
  namespace: actions-runner-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::$AWS_ACCOUNT_ID:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY 
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-ajh
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      organization: org
      serviceAccountName: sre-actions-runner-system-sa-ajh
      env:
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::$AWS_ACCOUNT_ID:role/eksctl-sre-eks-test-addon-iamserviceaccount-Role1-1244USHBUKLEY
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      volumeMounts:
      - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
        name: aws-iam-token 
        readOnly: true
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      securityContext:
        fsGroup: 27
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: hra-1-ajh
  namespace: actions-runner-system
spec:
  scaleDownDelaySecondsAfterScaleUp: 10
  scaleTargetRef:
    name: runner-ajh
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
    - test-repo

After that it was just normal the AWS assume role setup.

gregorygtseng · 2020-09-04T17:40:07Z

@callum-tait-pbx Oh I see, you didn't mount the service token yourself which is why you need the annotation. Glad that combination worked out as well. Check the running pod spec to confirm.

Nuru · 2020-10-21T00:00:49Z

I believe this is similar to jenkinsci/kubernetes-operator#361

When you add the eks.amazonaws.com/role-arn annotation to the service account, EKS automatically adds things to the Pod, which makes the Pod not look like what the controller is expecting, so the controller tries to fix it by deleting and recreating the Pod. The workaround is to tell the Controller to expect exactly what EKS is going to add.

I believe the better solution is for the Controller to deploy Deployments, not Pods (#133).

droidpl · 2020-11-13T18:14:28Z

Thanks for all your guidance with this @callum-tait-pbx and @gregorygtseng . I got all of this also working to share account credentials through the AssumeWebIdentity procedure.

My deployment file is almost a clone of yours 😸

mumoshu · 2021-04-25T07:37:42Z

We already have the documentation for this at https://github.com/summerwind/actions-runner-controller#using-eks-iam-role-for-service-accounts so it should be straightforward today. Closing as resolved!

…nts (#16)

mumoshu added the question Further information is requested label Mar 22, 2020

Nuru mentioned this issue Oct 21, 2020

Runners should be deployed in standard Deployments and ReplicaSets #133

Closed

callum-tait-pbx mentioned this issue Nov 17, 2020

Feature: Natively Support IRSA on EKS Properly #200

Closed

mumoshu closed this as completed Apr 25, 2021

TingluoHuang pushed a commit that referenced this issue Jan 12, 2023

Setup a wrapper class that can generate multiple Actions Service clie…

db6d523

…nts (#16)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runner Pod can't start up with ServiceAccount on EKS 1.15.10 #16

Runner Pod can't start up with ServiceAccount on EKS 1.15.10 #16

missedone commented Mar 21, 2020

mumoshu commented Mar 21, 2020

missedone commented Mar 21, 2020

mumoshu commented Mar 21, 2020

mumoshu commented Mar 21, 2020

alexandrst88 commented Mar 23, 2020

nachomillangarcia commented Jul 8, 2020

callum-tait-pbx commented Aug 26, 2020 •

edited

Loading

gregorygtseng commented Sep 2, 2020

callum-tait-pbx commented Sep 3, 2020 •

edited

Loading

gregorygtseng commented Sep 3, 2020

callum-tait-pbx commented Sep 3, 2020 •

edited

Loading

gregorygtseng commented Sep 3, 2020 •

edited

Loading

callum-tait-pbx commented Sep 4, 2020 •

edited

Loading

gregorygtseng commented Sep 4, 2020

Nuru commented Oct 21, 2020

droidpl commented Nov 13, 2020

mumoshu commented Apr 25, 2021

Runner Pod can't start up with ServiceAccount on EKS 1.15.10 #16

Runner Pod can't start up with ServiceAccount on EKS 1.15.10 #16

Comments

missedone commented Mar 21, 2020

mumoshu commented Mar 21, 2020

missedone commented Mar 21, 2020

mumoshu commented Mar 21, 2020

mumoshu commented Mar 21, 2020

alexandrst88 commented Mar 23, 2020

nachomillangarcia commented Jul 8, 2020

callum-tait-pbx commented Aug 26, 2020 • edited Loading

gregorygtseng commented Sep 2, 2020

callum-tait-pbx commented Sep 3, 2020 • edited Loading

gregorygtseng commented Sep 3, 2020

callum-tait-pbx commented Sep 3, 2020 • edited Loading

gregorygtseng commented Sep 3, 2020 • edited Loading

callum-tait-pbx commented Sep 4, 2020 • edited Loading

gregorygtseng commented Sep 4, 2020

Nuru commented Oct 21, 2020

droidpl commented Nov 13, 2020

mumoshu commented Apr 25, 2021

callum-tait-pbx commented Aug 26, 2020 •

edited

Loading

callum-tait-pbx commented Sep 3, 2020 •

edited

Loading

callum-tait-pbx commented Sep 3, 2020 •

edited

Loading

gregorygtseng commented Sep 3, 2020 •

edited

Loading

callum-tait-pbx commented Sep 4, 2020 •

edited

Loading