set k8s registry QPS to match MAX_PODS #1801
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -524,6 +524,8 @@ echo "$(jq --arg mebibytes_to_reserve "${mebibytes_to_reserve}Mi" --arg cpu_mill | |||
|
|
|||
| if [[ "$USE_MAX_PODS" = "true" ]]; then | |||
| echo "$(jq ".maxPods=$MAX_PODS" $KUBELET_CONFIG)" > $KUBELET_CONFIG | |||
| #set registryPullQPS to match MAX_PODS to prevent startup problems when nodes failover | |||
| echo "$(jq --arg MAX_PODS $MAX_PODS '.+= {"registryPullQPS":$MAX_PODS}' $KUBELET_CONFIG)" > $KUBELET_CONFIG | |||
There was a problem hiding this comment.
I think it makes sense to increase this beyond 5; but I don't think we want to go all the way to MAX_PODS -- that's a very large value on many instance types. Have you tested a more moderate increase, something like 10+15burst (up from 5+10burst)?
There was a problem hiding this comment.
I actually just set it to 0 (disabled). containerd times out commonly for me starting up 50-ish pods at once so I've also started to bump runtimeRequestTimeout.
I definitely think this number should be dynamic and I was suggesting max_pods as a proxy since, if a pod is "full" and it fails, the next one to boot up will potentially get all of those pods assigned.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
When nodes failover in EKS, regardless of their size, the default RegistryPullQPS of 5 highly limits their ability to startup cleanly when running a cluster with more than a few pods.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Testing Done
node failovers/drains on my clusters constantly fail. Bumping the QPS with user-data solves it but this, I think, is a better default.
See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.