nodeadm: retry IMDS 404 errors

[ndbaker1]

Issue #, if available:

Description of changes:

nodeadm fails to make IMDS calls when the instance's credentials propagate slower, causing IMDS to return an error indicating no iam credentials were provided for an instance and a 404 is returned.

It now checks for this error message and counts it as retryable

Aug 28 19:12:30 localhost nodeadm[1491]: {"level":"info","ts":1724872350.0166261,"caller":"init/init.go:148","msg":"Fetching instance details.."}
Aug 28 19:12:30 localhost nodeadm[1491]: SDK 2024/08/28 19:12:30 DEBUG attempting waiter request, attempt count: 1
Aug 28 19:12:30 localhost nodeadm[1491]: SDK 2024/08/28 19:12:30 DEBUG request failed with unretryable error http response error StatusCode: 404, request to EC2 IMDS failed

cloud-init showing IMDS resolving ~2 second later

2024-08-28 19:12:32,448 - url_helper.py[DEBUG]: Read from http://169.254.169.254:80/2021-03-23/meta-data/instance-id (200, 19b) after 1 attempts

other notable changes:

• removed all direct uses of the aws imds client besides in the internal helper implementation

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing Done

we have a test case https://github.com/awslabs/amazon-eks-ami/blob/71aa1082300050221bb940abf14f0ba635fc59eb/nodeadm/test/e2e/cases/imds-timeouts/run.sh, meant to show that nodeadm will succeed if IMDS comes up in the middle of execution

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

[ndbaker1]

/ci

[github-actions]

@ndbaker1 roger that! I've dispatched a workflow. 👍

[github-actions]

@ndbaker1 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2023	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅

[mattcjo]

@nbaker1 Quick update on root cause. The node successfully authenticated with IMDS, but the resource it was trying to access was missing. Retry logic is still an appropriate solution here.

[ndbaker1]

rerunning CI as we reverted back to the original approach

/ci

[github-actions]

@ndbaker1 roger that! I've dispatched a workflow. 👍

[github-actions]

@ndbaker1 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2023	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅