Support for IAM Roles for Service Accounts for EKS

[sudip-moengage]

Is your feature request related to a problem? Please describe.
It is not a problem for me, for now I'll create access key and secret but would like to have support for IAM roles, so that only pods have needed iam role for the S3 output pligin or cloudwatch output plugin

Additional context
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

[ahma]

I think the S3 output now has all the needed properties. Since the last release we added all the missing auth configs to cloudwatch output #273 .You can change the fluentd service account if you want. I'm going to close this ticket, but if you are missing anything please reopen it, and let us know.

[sudip-moengage]

@ahma I couldn't test it, but I checked #273, I'm not proficient with go, but I can see support for assume instance role is there, which the pod can inherit from node.

But this issue is about kubernetes service account integration with AWS IAM roles. In short we do following:

• In AWS create roles with following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "${OIDC_ARN}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_URL}:sub": "system:serviceaccount:${NAMESPACE}:${SA_NAME}"
        }
      }
    }
  ]
}

• In kubernetes we need to create a service account and annotate it with the role and use this service account in pods.

Then EKS automatically adds AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE (volume mount) env variable to pod. So, we need support for assuming roles via web identity token.

[ahma]

Thanks @sudip-moengage , I will look into this. -> reopen

[ahma]

@sudip-moengage In this case you can annotate the SA If you annotate the fluentd/fluentdbit resources Docs

[sudip-moengage]

@ahma Yes, I can annotate but, the SDK fluentd uses, does it support sts:AssumeRoleWithWebIdentity?

[bonifaido]

Hi @sudip-moengage,

the root cause for this issue is that the AWS Ruby SDK in the latest fluentd-s3-plugin doesn't support those environment variables injected by EKS in this case (AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE).

Please follow fluent/fluent-plugin-s3#301 for further details. Once that plugin gets updated with a more up-to-date AWS Ruby SDK this will work automagically in more recent release of that plugin.

We can't solve this issue from the logging-operator, so I'm closing this now.

[bonifaido]

@sudip-moengage just submitted fluent/fluent-plugin-s3#305 to resolve the issue upstream, I hope it gets merged soon enough 🙂

[gregorygtseng]

Correct me if I'm wrong, but I don't think the CRD supports annotating the ServiceAccount for fluentd; it only allows annotations on the fluentd pods themselves, or naming a different SA.

[kostyaplis]

@gregorygtseng seems that you are right. Having annotation set as fluentd.annotations annotates a fluentd pod, but not service account. Is there a way to let operator set SA annotation?

[pepov]

no, currently you cannot

[kostyaplis]

@pepov thanks for prompt reply
Shall we reopen this issue then?
I confirm that SA annotation is the only missing feature to fully support IRSA for EKS.

[prein]

Need help understanding what the conclusion is here. Is assuming roles via web identity token supported? Does it work for anyone? If so, would you mind sharing an example working config?

[pepov]

Although I don't have a complete example to show you, but the conclusion is, that yes it should work.

Once you go through the steps of setting up IAM Roles for Service Accounts there is one item that needs to be done on the operator side, which is annotating the fluentd service account.

For that you need a config similar to this in your logging resource:

spec:
  fluentd:
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role

This will merge the above annotation into the existing ones.