Fix index pattern when querying ES and condition when searching logs #3765

belimawr · 2023-11-14T12:26:22Z

What does this PR do?

Two issues were causing flakiness on integration tests and are fixed by this PR:

The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless as well as make it more specific to the indexes/data streams we want to query
findESDocs did not wait for the data to be indexed, only to a successful query on ES. In some cases the documents the test wanted were not indexed yet, leading to 0 documents being returned and the test failing with no error in the logs/diagnostics. This is fixed by waiting for a document count > 0 and no error.

Why is it important?

It fixes flaky integration tests.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
~~- [ ] I have added an entry in ./changelog/fragments using the changelog tool~~
I have added an integration test or an E2E test

~~## Author's Checklist~~

How to test this PR locally

Run the integration test that was failing with both serverless and stateful

STACK_PROVISIONER=serverless TEST_PLATFORMS="linux/amd64" SNAPSHOT=true mage -v integration:single TestLogIngestionFleetManaged
mage integration:clean
TEST_PLATFORMS="linux/amd64" SNAPSHOT=true mage -v integration:single TestLogIngestionFleetManaged

It is needed to clean the deployments because at the moment the integration test framework does not keep track of the type of stack it deployed, more information: #3756

Related issues

~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

elasticmachine · 2023-11-14T12:26:25Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

mergify · 2023-11-14T12:26:58Z

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-11-14T13:34:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-20T09:04:05.333+0000
Duration: 14 min 44 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

pkg/testing/tools/estools/elasticsearch.go

testing/integration/logs_ingestion_test.go

belimawr · 2023-11-15T17:54:44Z

The failure seems to be caused by: #3664

belimawr · 2023-11-16T11:35:58Z

buildkite test this

testing/integration/endpoint_security_test.go

The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless.

findESDocs now ensures the function returns at least one document. Debug logs are also removed.

elastic-sonarqube · 2023-11-27T15:07:59Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

belimawr · 2023-11-27T16:41:28Z

Folks, this PR is finally (well, hopefully) ready for a final review. I found another source of flakiness and fixed it in my last commit ff4edf2.

blakerouse

Looks good!

…3765) Two issues were causing flakiness on integration tests and are fixed by this PR: 1. The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless as well as make it more specific to the indexes/data streams we want to query 2. `findESDocs` did not wait for the data to be indexed, only to a successful query on ES. In some cases the documents the test wanted were not indexed yet, leading to 0 documents being returned and the test failing with no error in the logs/diagnostics. This is fixed by waiting for a document count > 0 and no error. (cherry picked from commit a03aa9c) # Conflicts: # pkg/testing/tools/estools/elasticsearch.go # testing/integration/logs_ingestion_test.go

…on when searching logs (#3829) Two issues were causing flakiness on integration tests and are fixed by this PR: 1. The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless as well as make it more specific to the indexes/data streams we want to query 2. `findESDocs` did not wait for the data to be indexed, only to a successful query on ES. In some cases the documents the test wanted were not indexed yet, leading to 0 documents being returned and the test failing with no error in the logs/diagnostics. This is fixed by waiting for a document count > 0 and no error. (cherry picked from commit a03aa9c) # Conflicts: # pkg/testing/tools/estools/elasticsearch.go # testing/integration/logs_ingestion_test.go The `add_cloud_metadata` and some other `add_*_metadata` processors are expected to log some errors if they cannot fetch the necessary information. It is normal to find their error logs in pretty much any deployment, some examples: - When Docker is not installed/running `add_docker_metadata` will log some errors - When the Elastic-Agent is deployed in a non-cloud VM `add_cloud_metadata` will log some errors. This commit removes all those processors from the queries for log errors, as they're expected. Add a 2h timeout for `go test` when run on remote hosts. --------- Co-authored-by: Tiago Queiroz <[email protected]>

belimawr added Team:Elastic-Agent Label for the Agent team skip-changelog labels Nov 14, 2023

belimawr requested a review from a team as a code owner November 14, 2023 12:26

belimawr requested review from AndersonQ and michalpristas November 14, 2023 12:26

belimawr added backport-v8.11.0 Automated backport with mergify and removed backport-v8.11.0 Automated backport with mergify labels Nov 14, 2023

mergify bot assigned belimawr Nov 14, 2023

mergify bot added the backport-skip label Nov 14, 2023

belimawr mentioned this pull request Nov 14, 2023

Revert "Support flattened data_stream.* fields" #3743

Closed

belimawr force-pushed the fix-integration-tests branch 2 times, most recently from bb34c4d to 94b2ffe Compare November 14, 2023 16:57

cmacknz reviewed Nov 14, 2023

View reviewed changes

pkg/testing/tools/estools/elasticsearch.go Show resolved Hide resolved

cmacknz reviewed Nov 14, 2023

View reviewed changes

testing/integration/logs_ingestion_test.go Show resolved Hide resolved

cmacknz reviewed Nov 14, 2023

View reviewed changes

testing/integration/logs_ingestion_test.go Outdated Show resolved Hide resolved

cmacknz reviewed Nov 14, 2023

View reviewed changes

testing/integration/logs_ingestion_test.go Outdated Show resolved Hide resolved

belimawr force-pushed the fix-integration-tests branch from 34790b7 to 621b857 Compare November 15, 2023 09:24

belimawr force-pushed the fix-integration-tests branch from eb9d8f3 to dbb399d Compare November 20, 2023 09:03

blakerouse requested changes Nov 20, 2023

View reviewed changes

testing/integration/endpoint_security_test.go Outdated Show resolved Hide resolved

belimawr force-pushed the fix-integration-tests branch 2 times, most recently from 4603baf to 31b8184 Compare November 23, 2023 14:03

belimawr added 4 commits November 27, 2023 09:05

Fix index pattern when querying ES

76104ad

The pattern used to query ES was not working on serverless, this commit updates it to a pattern that works on both stateful and serverless.

increase query time and add debug

ef03e68

more debug

10aabb3

remove debug

867a3d8

belimawr added 7 commits November 27, 2023 09:06

try fixing endpoint tests by isolating them

999d579

fix batchID length, revert endpoint test changes

a45f70a

only query Elastic-Agent logs datastream

bbfb319

revert eventually time

1dd88ce

last index/ds name update

7f273ff

improve test error message

d610f90

debug logs

a081090

belimawr force-pushed the fix-integration-tests branch 2 times, most recently from 787f635 to c1e4f96 Compare November 27, 2023 14:20

findESDocs ensures non-zero documents

ff4edf2

findESDocs now ensures the function returns at least one document. Debug logs are also removed.

belimawr force-pushed the fix-integration-tests branch from c1e4f96 to ff4edf2 Compare November 27, 2023 14:37

belimawr changed the title ~~Fix index pattern when querying ES~~ Fix index pattern when querying ES and condition when searching logs Nov 27, 2023

belimawr requested review from blakerouse and cmacknz November 27, 2023 16:41

cmacknz approved these changes Nov 27, 2023

View reviewed changes

blakerouse approved these changes Nov 27, 2023

View reviewed changes

belimawr merged commit a03aa9c into elastic:main Nov 28, 2023
9 checks passed

belimawr deleted the fix-integration-tests branch November 28, 2023 09:03

belimawr added the backport-v8.11.0 Automated backport with mergify label Nov 28, 2023

mergify bot mentioned this pull request Nov 28, 2023

[8.11](backport #3765) Fix index pattern when querying ES and condition when searching logs #3829

Merged

rdner mentioned this pull request Mar 27, 2024

TestLogIngestionFleetManaged/Monitoring_logs_are_shipped is flaky #3741

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix index pattern when querying ES and condition when searching logs #3765

Fix index pattern when querying ES and condition when searching logs #3765

belimawr commented Nov 14, 2023 •

edited

Loading

elasticmachine commented Nov 14, 2023

mergify bot commented Nov 14, 2023

elasticmachine commented Nov 14, 2023 •

edited

Loading

Build stats

belimawr commented Nov 15, 2023

belimawr commented Nov 16, 2023

elastic-sonarqube bot commented Nov 27, 2023

belimawr commented Nov 27, 2023

blakerouse left a comment

Fix index pattern when querying ES and condition when searching logs #3765

Fix index pattern when querying ES and condition when searching logs #3765

Conversation

belimawr commented Nov 14, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Questions to ask yourself

elasticmachine commented Nov 14, 2023

mergify bot commented Nov 14, 2023

elasticmachine commented Nov 14, 2023 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

belimawr commented Nov 15, 2023

belimawr commented Nov 16, 2023

elastic-sonarqube bot commented Nov 27, 2023

belimawr commented Nov 27, 2023

blakerouse left a comment

Choose a reason for hiding this comment

belimawr commented Nov 14, 2023 •

edited

Loading

elasticmachine commented Nov 14, 2023 •

edited

Loading