Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.11](backport #3765) Fix index pattern when querying ES and condition when searching logs #3829

Merged
merged 4 commits into from
Nov 30, 2023

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Nov 28, 2023

Why so many changes on this backport?

Even though this PR is a backport, there were quite some changes to the integration tests and tooling in main, because of conflicts I took the path of least resistance and copied the new tests and some of the tooling around them, in the end we get some extra test coverage.

Another change I had to make was to introduce 9faef34 excluding some add_*_metadata processors from the default query that searches for error logs. add_cloud_metadata was always logging some errors and I have seen other similar processors logging errors almost every time I run the Elastic-Agent. So I decided to exclude them all by default.

The automated message

fixes #3831

This is an automatic backport of pull request #3765 done by Mergify.
Cherry-pick of a03aa9c has failed:

On branch mergify/bp/8.11/pr-3765
Your branch is up to date with 'origin/8.11'.

You are currently cherry-picking commit a03aa9cf19.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   pkg/testing/tools/estools/elasticsearch.go
	deleted by us:   testing/integration/logs_ingestion_test.go

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

@mergify mergify bot requested a review from a team as a code owner November 28, 2023 09:08
@mergify mergify bot added the backport label Nov 28, 2023
@mergify mergify bot removed the request for review from a team November 28, 2023 09:08
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Nov 28, 2023
@elasticmachine
Copy link
Contributor

elasticmachine commented Nov 28, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-29T08:56:31.197+0000

  • Duration: 15 min 25 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages.

  • run integration tests : Run the Elastic Agent Integration tests.

  • run end-to-end tests : Generate the packages and run the E2E Tests.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@pierrehilbert pierrehilbert requested review from belimawr and removed request for michalpristas and pchila November 28, 2023 11:21
@pierrehilbert pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Nov 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

…3765)

Two issues were causing flakiness on integration tests and are fixed by this PR:
1. The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless as well as make it more specific to the indexes/data streams we want to query
2. `findESDocs` did not wait for the data to be indexed, only to a successful query on ES. In some cases the documents the test wanted were not indexed yet, leading to 0 documents being returned and the test failing with no error in the logs/diagnostics. This is fixed by waiting for a document count > 0 and no error.

(cherry picked from commit a03aa9c)

# Conflicts:
#	pkg/testing/tools/estools/elasticsearch.go
#	testing/integration/logs_ingestion_test.go
The `add_cloud_metadata` and some other `add_*_metadata` processors
are expected to log some errors if they cannot fetch the necessary
information. It is normal to find their error logs in pretty much any
deployment, some examples:
 - When Docker is not installed/running `add_docker_metadata` will log
 some errors
 - When the Elastic-Agent is deployed in a non-cloud VM
 `add_cloud_metadata` will log some errors.

This commit removes all those processors from the queries for log
errors, as they're expected.
@AndersonQ
Copy link
Member

buildkite test this

Add a 2h timeout for `go test` when run on remote hosts.
Copy link

SonarQube Quality Gate

Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@belimawr belimawr merged commit a10d713 into 8.11 Nov 30, 2023
12 checks passed
@belimawr belimawr deleted the mergify/bp/8.11/pr-3765 branch November 30, 2023 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants