Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x](backport #5699) Only watch metadata for ReplicaSets in K8s provider #5734

Merged
merged 1 commit into from
Oct 9, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Oct 8, 2024

What does this PR do?

Use a metadata watcher for ReplicaSets in the K8s provider. The only data we need from ReplicaSets are their name and OwnerReferences, which are used to connect Pods to Deployments and DaemonSets.

Why is it important?

This significantly reduces both memory used to store ReplicaSet data, and temporary allocations to process update events from the API Server. See the linked issue for more detailed information.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added an entry in ./changelog/fragments using the changelog tool

How to test this PR locally

Showing that the change doesn't cause a regression requires starting agent in K8s and enabling deployment metadata in the K8s provider, and then using it in a templated input definition - this is simplest with filebeat.

Demonstrating the memory consumption improvement is more involved - you need to create a significant amount of ReplicaSets in the cluster. I saw a noticable difference at ~7500, see #5623 for more details.

Related issues


This is an automatic backport of pull request #5699 done by [Mergify](https://mergify.com).

@mergify mergify bot requested review from a team as code owners October 8, 2024 08:04
@mergify mergify bot added the backport label Oct 8, 2024
@mergify mergify bot requested review from MichaelKatsoulis and constanca-m and removed request for a team October 8, 2024 08:04
@mergify mergify bot requested review from blakerouse and swiatekm and removed request for a team October 8, 2024 08:04
@swiatekm
Copy link
Contributor

swiatekm commented Oct 8, 2024

Blocked by #5702

Use a metadata informer for ReplicaSets.
Use a transform function to drop all data except for the owner
references, which we need to find the Deployment name.

(cherry picked from commit 05f9e81)
@swiatekm swiatekm enabled auto-merge (squash) October 9, 2024 09:59
@swiatekm swiatekm merged commit 3c8e25a into 8.x Oct 9, 2024
14 checks passed
@swiatekm swiatekm deleted the mergify/bp/8.x/pr-5699 branch October 9, 2024 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants