Add serverless beats tests #3658
Conversation
| - label: "Serverless Beats Tests" | ||
| key: "serverless-beats-integration-tests" | ||
| command: ".buildkite/scripts/steps/beats_tests.sh" | ||
| if: "build.env('CRON') == 'yes'" |
There was a problem hiding this comment.
Where is the cron schedule controlled? Can you link to it here?
There was a problem hiding this comment.
So, I was going to add it once the PR was merged, but the control panel for it is here: https://buildkite.com/elastic/elastic-agent/settings/schedules
There was a problem hiding this comment.
Add a link to that as a comment right above this so future us can find it easily
|
Do you have a link to the Beats tests passing on Buildkite, since they aren't running as part of the PR automation? |
| fi | ||
|
|
||
|
|
||
| echo "testing auditbeat..." |
There was a problem hiding this comment.
We can skip testing auditbeat since our team doesn't maintain it. None of the code we are testing is really Beat specific regardless.
| mkdir -p build | ||
| cd build | ||
|
|
||
| git clone [email protected]:elastic/beats.git |
There was a problem hiding this comment.
Why do we need to clone and build Beats?
Why don't we just download the latest SNAPSHOT image using the artifacts API?
There was a problem hiding this comment.
+1
If we need to clone the repo, maybe we can get by with a partial clone (for example git clone --filter=tree:0 [email protected]:elastic/beats.git) to improve speed ?
|
Your sample test run doesn't contain any Beat logs. If these tests fail how do we debug them? There are no test artifacts generated either https://buildkite.com/elastic/elastic-agent/builds/4490#018b87a1-0854-4559-b675-20e3033f15e4 So there is no test report generated to make the failures easier to see. It also looks like it is running every other test. Is that the intent for the scheduled job? I would have it only run the serverless Beats test so it isn't going to fail for any other random reason. |
There was a problem hiding this comment.
A couple of comments left about the pipeline step script and the mage target.
Overall code is ok, I am not too fond of the mixing of beats serverless and elastic agent integration tests being guarded only by a condition on the TestSuite and a mage target that would allow to run those tests against a stateful stack (not sure that we have a usecase for that).
| mkdir -p build | ||
| cd build | ||
|
|
||
| git clone [email protected]:elastic/beats.git |
There was a problem hiding this comment.
+1
If we need to clone the repo, maybe we can get by with a partial clone (for example git clone --filter=tree:0 [email protected]:elastic/beats.git) to improve speed ?
| if ! run_test_for_beat metricbeat; then | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Nitpick: the script would fail anyway if run_test_for_beat metricbeat returns != 0 thanks to the set -e above
This means we don't need to handle the return code unless we wanna try to run all the test and return a correct exit status at the end (similar to what's done on lines 53-54)
There was a problem hiding this comment.
ah, good point!
| // or more contributor license agreements. Licensed under the Elastic License; | ||
| // you may not use this file except in compliance with the Elastic License. | ||
|
|
||
| //go:build integration |
There was a problem hiding this comment.
@cmacknz my understanding is that today we have the guard in the SetupSuite() below (lines 71-73):
if runner.testbeatName == "elastic-agent" {
runner.T().Skipf("tests must be run against a beat, not elastic-agent")
}
@fearful-symmetry is this what's stopping these tests from failing when launching elastic-agent integration tests or am I missing something ?
|
@pchila not sure what you mean? The |
|
SonarQube Quality Gate
|
What does this PR do?
This is (hopefully) the final stage of getting serverless testing working. This Adds 17 serverless tests, as well as a mage role to run the tests. Assuming you have a beats binary built, you can run
mage integration:testBeatServerless metricbeatThis does not include any buildkite steps to run the tests; I'm not totally sure how we want to schedule them, and if that should be part of this PR.
Why is it important?
We need serverless tests.
Checklist
./changelog/fragmentsusing the changelog tool