Karpenter IAM policy version 1.0.2 is not backwards compatible with 0.37.3

[dcherniv]

Description

Observed Behavior:
When policy is updated to version 1.0.2 and karpenter at version 0.37.3, karpenter starts failing with errors:
#6847 (comment)
In order to enable smooth upgrade path, the policy needs to be compatible, otherwise you run into chicken/egg problems. You upgrade karpenter to 1.0.2 but it doesn't work the old policy and karpenter 0.37.x doesn't work with the new polciy
This is mentioned here at step 8 https://karpenter.sh/v1.0/upgrading/v1-migration/#upgrade-procedure
but it's not a good solution because webhooks that supposed to convert the resources on 1.0.x straight up don't work
#6982
We should keep the policy backwards compatible.
Expected Behavior:
Policy to be backwards compatible for a time to allow users to migrate.

Reproduction Steps (Please include YAML):
Upgrade the policy to 1.0.2 version while karpenter is on 0.37.3 version

Versions:

• Chart Version: 0.37.3, 1.0.2
• Kubernetes Version (kubectl version): 1.29+ eks

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

We should keep the policy backwards compatible

I'm not sure that the right solution here is to make the policy backwards compatible because we don't want a bunch of old permissions hanging around from old versions just because users might be going between versions. In my mind, the right way to handle this is probably to just deploy two policies and attach them to the role and then remove the old policy when you are fully on the new version.

Either that or we should call out the permissions that need to be added more explicitly in the upgrade guide. In either case, I think this is most likely a docs issue with the v1 upgrade guide. Have any thoughts on it?

[dcherniv]

Either that or we should call out the permissions that need to be added more explicitly in the upgrade guide. In either case, I think this is most likely a docs issue with the v1 upgrade guide. Have any thoughts on it?

I agree that docs should be clearer on the step 8 in v1 migration guide https://karpenter.sh/v1.0/upgrading/v1-migration/
But that is until you have to rollback to 0.37.3 like i did (many times due to that stale nodeclaims issue) But even knowing that policy is incompatible would go a long way. I was completely blindsided for one

[jonathan-innis]

I was completely blindsided for one

Makes sense. We can take up a fix to make this clearer.