Introduce RequestEviction feature for evicting pods in background (KEP-1397) #1466

ingvagabund · 2024-07-14T17:55:41Z

Implementation of #1354.

When the feature is enabled each pod with descheduler.alpha.kubernetes.io/request-evict-only annotation will have the eviction API error examined for a specific error code/reason and message. If matched eviction of such a pod will be interpreted as initiation of an eviction in background.

The first version of the feature does not provide any way to configure a timeout for pending evictions in background. Neither a timeout for how often a pending evictions in background are checked for timeouts. Both set to 10 minutes by default. The default values can be adjusted based on provided feedback.

TODO:

evictions in background are respested
unit tests
feature gating implemented (EvictionsInBackground as a new feature gate)
e2e integration with kubevirt
feature configuration (currently none)

Refactorings before this PR can be merged:

ingvagabund · 2024-07-18T11:00:24Z

Compared to virt-handler produced by my local kind these are the logs produced by the upstream CI:

{"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance default/vmi-masquerade-3","pos":"vm.go:1712","reason":"server error. command SyncVMI failed: \"LibvirtError(Code=38, Domain=0, Message='Unable to create tap device tap0: Permission denied')\"","timestamp":"2024-07-17T15:01:56.211181Z"}

Could be kubevirt does not like combination of running kind in pod with dind enabled.

ingvagabund · 2024-07-19T07:22:21Z

/retest-required

ingvagabund · 2024-09-02T14:36:36Z

/retest-required

pkg/descheduler/descheduler_test.go

pkg/descheduler/evictions/evictions.go

fanhaouu · 2024-09-23T11:25:15Z

pkg/descheduler/evictions/evictions.go

@@ -210,13 +569,36 @@ func evictPod(ctx context.Context, client clientset.Interface, pod *v1.Pod, poli
 		},
 		DeleteOptions: deleteOptions,
 	}
-	err := client.PolicyV1().Evictions(eviction.Namespace).Evict(ctx, eviction)
+	err := pe.client.PolicyV1().Evictions(eviction.Namespace).Evict(ctx, eviction)


Have you considered adding a similar retry mechanism?

retriable := func(err error) bool {
return apierrors.IsTooManyRequests(err) || apierrors.IsInternalError(err) || apierrors.IsServerTimeout(err) || apierrors.IsUnexpectedServerError(err)
}
err = retry.OnError(retry.DefaultRetry, retriable, func() error {
return evictPod(ctx, pe.client, pod, pe.directEvict)
})

I believe we discussed this in the past. Though, I have not seen anyone reporting this as annoying. We can always update the code. Nevertheless, if the returned errors are returned again for some period of time, retrying will not help. If only few evictions return the error codes the descheduler will try to evict another pod. All the pods are still treated as cattle. So the impact is usually negligible.

Yes, the impact is not significant. For example, the rescheduling policy uses the method podutil.SortPodsBasedOnPriorityLowToHigh(removablePods) to sort the pods. Lower-priority pods at the front may fail to be evicted due to network issues, resulting in the eviction of higher-priority pods later, even though all these pods are evictable.

You are right. This could be a suboptional case. Too many higher priority pods getting evicted instead of lower priority ones. Nevertheless, this deserves a separate PR/change. Retries are outside the scope of this.

fanhaouu · 2024-09-23T11:40:30Z

No obvious bugs were found; everything except the commented code looks good to me.

ingvagabund · 2024-10-07T11:30:34Z

/retest-required

k8s-ci-robot · 2024-10-07T12:11:28Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

atiratree · 2024-10-09T09:28:59Z

cmd/descheduler/app/options/options.go

@@ -45,6 +50,10 @@ type DeschedulerServer struct {
 	SecureServing  *apiserveroptions.SecureServingOptionsWithLoopback
 	DisableMetrics bool
 	EnableHTTP2    bool
+	// FeatureGates enabled by the user


Btw, the approach used by upstream is via ComponentGlobalsRegistry. Might be good to consider if it makes sense to introduce that.

atiratree · 2024-10-09T09:35:43Z

cmd/descheduler/app/server.go

@@ -115,7 +116,13 @@ func NewDeschedulerCommand(out io.Writer) *cobra.Command {
 }

 func Run(ctx context.Context, rs *options.DeschedulerServer) error {
-	err := tracing.NewTracerProvider(ctx, rs.Tracing.CollectorEndpoint, rs.Tracing.TransportCert, rs.Tracing.ServiceName, rs.Tracing.ServiceNamespace, rs.Tracing.SampleRate, rs.Tracing.FallbackToNoOpProviderOnError)
+	err := features.DefaultMutableFeatureGate.SetFromMap(rs.FeatureGates)


the initializations would be better placed in the apply logic

atiratree · 2024-10-09T09:43:30Z

pkg/descheduler/descheduler.go

 		eventRecorder,
+		podInformer,
+		rs.DefaultFeatureGates,


Does it make sense to pass the feature gates as an argument everywhere? Compared to the global variable.

I was keeping in mind the case where the descheduling framework is run as a simulator. E.g. in case two or more framework instances are created in the same run/binary.

pkg/descheduler/descheduler.go

atiratree · 2024-10-09T10:08:52Z

pkg/descheduler/evictions/evictions.go

+						return
+					}
+					// Ignore any pod that does not have eviction in progress
+					if _, exists := newPod.Annotations[EvictionInProgressAnnotationKey]; !exists {


how can we get to this line and delete pods that lost their annotation, when we do

if newPod.Annotations == nil { return } // Ignore pod's that are not subject to an eviction in background if _, exists := newPod.Annotations[EvictionRequestAnnotationKey]; !exists { return }

just above?

If a pod looses its annotation there's no way of knowing whether the annotation was removed by accident or intentionally. E.g. a live migration is no longer needed since some scoring mechanism decided it's more costly to perform a live migration than to drop VM's state and re-create it.

atiratree · 2024-10-09T10:49:21Z

pkg/descheduler/evictions/evictions.go

+
+type evictionRequestItem struct {
+	pod               *v1.Pod
+	assumed           bool


can we please make the name of the field more verbose?

Renamed to evictionAssumed

pkg/descheduler/evictions/evictions.go

atiratree · 2024-10-09T10:55:59Z

pkg/framework/profile/profile.go

@@ -314,7 +315,7 @@ func (d profileImpl) RunDeschedulePlugins(ctx context.Context, nodes []*v1.Node)
 			span.AddEvent("Plugin Execution Failed", trace.WithAttributes(attribute.String("err", status.Err.Error())))
 			errs = append(errs, fmt.Errorf("plugin %q finished with error: %v", pl.Name(), status.Err))
 		}
-		klog.V(1).InfoS("Total number of pods evicted", "extension point", "Deschedule", "evictedPods", d.podEvictor.TotalEvicted()-evicted)
+		klog.V(1).InfoS("Total number of evictions/requests", "extension point", "Deschedule", "evictedPods", d.podEvictor.TotalEvicted()-evicted, "evictionRequests", d.podEvictor.TotalEvictionRequests()-evictionRequests)


nit: it might be good to unify the logging and use keysAndValues everywhere

atiratree · 2024-10-09T11:05:41Z

pkg/descheduler/evictions/evictions.go

@@ -108,6 +423,46 @@ func (pe *PodEvictor) SetClient(client clientset.Interface) {
 	pe.client = client
 }

+func (pe *PodEvictor) evictionRequestsTotal() uint {


I forgot what was the consensus last time, but do we want to introduce new metrics for this feature? https://github.com/kubernetes-sigs/descheduler/blob/master/keps/1397-evictions-in-background/README.md#metrics

We do. Yet, not as part of this PR. I will have a follow up for all other changes.

atiratree · 2024-10-09T11:08:46Z

test/e2e/e2e_evictioninbackground_test.go

+		ObjectMeta: metav1.ObjectMeta{
+			Name: fmt.Sprintf("kubevirtvmi-%v", idx),
+			Annotations: map[string]string{
+				"descheduler.alpha.kubernetes.io/request-evict-only": "",


just to confirm, the feature depends only on the key, and not on the value? Would be good to add a little a bit of the documentation above EvictionRequestAnnotationKey and EvictionInProgressAnnotationKey

That is correct. +1 for a documentation. I still have a blog post to write.

When the feature is enabled each pod with descheduler.alpha.kubernetes.io/request-evict-only annotation will have the eviction API error examined for a specific error code/reason and message. If matched eviction of such a pod will be interpreted as initiation of an eviction in background.

k8s-ci-robot · 2024-11-12T15:53:35Z

@ingvagabund: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-descheduler-verify-master	`7d50a3b`	link	true	`/test pull-descheduler-verify-master`
pull-descheduler-test-e2e-k8s-master-1-29	`7d50a3b`	link	true	`/test pull-descheduler-test-e2e-k8s-master-1-29`
pull-descheduler-test-e2e-k8s-master-1-30	`7d50a3b`	link	true	`/test pull-descheduler-test-e2e-k8s-master-1-30`
pull-descheduler-test-e2e-k8s-master-1-31	`7d50a3b`	link	true	`/test pull-descheduler-test-e2e-k8s-master-1-31`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 14, 2024

k8s-ci-robot requested review from a7i and JaneLiuL July 14, 2024 17:56

ingvagabund force-pushed the eviction-in-background-code branch 17 times, most recently from 1c05dac to d022bf4 Compare July 17, 2024 14:54

ingvagabund force-pushed the eviction-in-background-code branch 6 times, most recently from 3210921 to 90a6b55 Compare July 19, 2024 07:18

ingvagabund mentioned this pull request Sep 2, 2024

WRKLDS-1454: Evictions in background downstream patch openshift/descheduler#118

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 9, 2024

ingvagabund force-pushed the eviction-in-background-code branch from 8203821 to edb0100 Compare September 17, 2024 09:44

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2024

ingvagabund mentioned this pull request Sep 23, 2024

[e2e] abstract common methods #1517

Merged

fanhaouu reviewed Sep 23, 2024

View reviewed changes

pkg/descheduler/descheduler_test.go Outdated Show resolved Hide resolved

fanhaouu reviewed Sep 23, 2024

View reviewed changes

pkg/descheduler/evictions/evictions.go Outdated Show resolved Hide resolved

fanhaouu reviewed Sep 23, 2024

View reviewed changes

pkg/descheduler/evictions/evictions.go Outdated Show resolved Hide resolved

fanhaouu reviewed Sep 23, 2024

View reviewed changes

pkg/descheduler/evictions/evictions.go Outdated Show resolved Hide resolved

fanhaouu reviewed Sep 23, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 1, 2024

ingvagabund force-pushed the eviction-in-background-code branch from edb0100 to 83d8ff9 Compare October 7, 2024 10:27

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 7, 2024

ingvagabund force-pushed the eviction-in-background-code branch 2 times, most recently from 79eebf1 to 9b5eae3 Compare October 7, 2024 11:21

ingvagabund force-pushed the eviction-in-background-code branch from 9b5eae3 to a2b5211 Compare October 7, 2024 11:39

ingvagabund added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 7, 2024

fanhaouu mentioned this pull request Oct 8, 2024

REQUEST: New membership for fanhaouu kubernetes/org#5208

Closed

11 tasks

atiratree reviewed Oct 9, 2024

View reviewed changes

ingvagabund added 4 commits November 12, 2024 12:22

Define EvictionsInBackground feature gate

923055b

bump(vendor)

798dbef

Create an Apply function

7d50a3b

ingvagabund mentioned this pull request Nov 12, 2024

DeschedulerServer: new Apply function for applying configuration #1542

Open

ingvagabund force-pushed the eviction-in-background-code branch from a2b5211 to 7d50a3b Compare November 12, 2024 15:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce RequestEviction feature for evicting pods in background (KEP-1397) #1466

Introduce RequestEviction feature for evicting pods in background (KEP-1397) #1466

ingvagabund commented Jul 14, 2024 •

edited

Loading

ingvagabund commented Jul 18, 2024

ingvagabund commented Jul 19, 2024

ingvagabund commented Sep 2, 2024

fanhaouu Sep 23, 2024

ingvagabund Sep 26, 2024

fanhaouu Sep 26, 2024

ingvagabund Sep 26, 2024

fanhaouu commented Sep 23, 2024

ingvagabund commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

atiratree Oct 9, 2024

atiratree Oct 9, 2024

atiratree Oct 9, 2024

ingvagabund Oct 10, 2024

atiratree Oct 9, 2024

ingvagabund Nov 12, 2024

atiratree Oct 9, 2024

ingvagabund Nov 12, 2024

atiratree Oct 9, 2024

atiratree Oct 9, 2024

ingvagabund Nov 12, 2024

atiratree Oct 9, 2024

ingvagabund Nov 12, 2024

k8s-ci-robot commented Nov 12, 2024

Introduce RequestEviction feature for evicting pods in background (KEP-1397) #1466

Are you sure you want to change the base?

Introduce RequestEviction feature for evicting pods in background (KEP-1397) #1466

Conversation

ingvagabund commented Jul 14, 2024 • edited Loading

ingvagabund commented Jul 18, 2024

ingvagabund commented Jul 19, 2024

ingvagabund commented Sep 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanhaouu commented Sep 23, 2024

ingvagabund commented Oct 7, 2024

k8s-ci-robot commented Oct 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 12, 2024

ingvagabund commented Jul 14, 2024 •

edited

Loading