Introduce upgrade tests from released versions #2955

pchila · 2023-06-28T08:56:13Z

What does this PR do?

This PR adds upgrade tests for both standalone and managed agent from some versions of the current and previous major release to the latest snapshot.

The tests will select:

up to 2 versions with the same major version as the current agent
up to 1 version from a previous major version
These versions are retrieved using the Artifact API and working backwards through the versions, excluding snapshots.
The up to part is because under certain circumstances (new major release, for example) we may not find enough versions fitting the criteria.

The tests TestStandaloneUpgrade and TestFleetManagedUpgrade will start a subtest (using t.Run()) for each version to be used as a starting point for the upgrade and then upgrade it to the latest version of agent.

Current limitations:

We are currently using the ArtifactFetcher so the starting elastic agent version all come from the artifact API. For all agent versions that allow the --skip-verify we use the packaged elastic agent as artifact for the upgrade via --sourceURI parameter

Why is it important?

We need to have automated tests for the common upgrade scenarios to detect upgrade issues as soon as possible in the development cycle.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
I have added an integration test or an E2E test

Author's Checklist

[ ]

How to test this PR locally

We can run the integration tests using
AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^(TestFleetManagedUpgrade|TestStandaloneUpgrade)$" mage integration:test
or if you want to run just a specific version upgrade scenario we can use (for example only standalone upgrade from 8.9.0 to 8.10.0-SNAPSHOT):
AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade/Upgrade_8.9.0_to_8.10.0-SNAPSHOT$"

Related issues

Requires Integration test - Fetcher must collect SHAs #3073

Use cases

Screenshots

Logs

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

elasticmachine · 2023-06-28T08:56:16Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

mergify · 2023-06-28T08:57:17Z

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-06-28T09:16:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-02T05:38:39.111+0000
Duration: 26 min 54 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6153
Skipped	31
Total	6184

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-06-28T09:17:06Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.718% (`77/78`)	👍
Files	68.498% (`187/273`)	👍
Classes	67.258% (`341/507`)	👍
Methods	53.796% (`1070/1989`)	👎 -0.027
Lines	40.034% (`12250/30599`)	👎 -0.006
Conditionals	100.0% (`0/0`)	💚

mergify · 2023-06-30T16:24:21Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hackathon/upgrade-integration-tests upstream/hackathon/upgrade-integration-tests
git merge upstream/main
git push upstream hackathon/upgrade-integration-tests

testing/integration/upgrade_test.go

pchila · 2023-07-13T15:26:04Z

buildkite test this

pierrehilbert · 2023-07-17T12:13:54Z

buildkite test this

mergify · 2023-07-17T12:29:12Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hackathon/upgrade-integration-tests upstream/hackathon/upgrade-integration-tests
git merge upstream/main
git push upstream hackathon/upgrade-integration-tests

pierrehilbert · 2023-07-20T06:53:09Z

buildkite test this

pchila · 2023-07-24T13:04:31Z

Last integration test didn't run any tests... There's probably an issue with the test enumeration...

pchila · 2023-08-01T15:39:13Z

buildkite test this

…previous major version

…latest previous major version

Co-authored-by: Blake Rouse <[email protected]>

## What does this PR do? This PR filters out versions that too new from agent upgrade integration tests, allowing backport of those test on release branches. This PR needs to backported together with #2955 onto 8.9 branch ## Why is it important? We want to have working upgrade tests with relevant versions also on release branches

## What does this PR do? This PR filters out versions that too new from agent upgrade integration tests, allowing backport of those test on release branches. This PR needs to backported together with #2955 onto 8.9 branch ## Why is it important? We want to have working upgrade tests with relevant versions also on release branches (cherry picked from commit 3de585e) # Conflicts: # testing/integration/upgrade_test.go

## What does this PR do? This PR adds upgrade tests for both standalone and managed agent from some versions of the current and previous major release to the latest snapshot. The tests will select: - up to 2 versions with the same major version as the current agent - up to 1 version from a previous major version These versions are retrieved using the Artifact API and working backwards through the versions, excluding snapshots. The _up to_ part is because under certain circumstances (new major release, for example) we may not find enough versions fitting the criteria. The tests `TestStandaloneUpgrade` and `TestFleetManagedUpgrade` will start a subtest (using `t.Run()`) for each version to be used as a starting point for the upgrade and then upgrade it to the latest version of agent. Current limitations: - We are currently using the ArtifactFetcher so the starting elastic agent version all come from the artifact API. For all agent versions that allow the `--skip-verify` we use the packaged elastic agent as artifact for the upgrade via `--sourceURI` parameter ## Why is it important? We need to have automated tests for the common upgrade scenarios to detect upgrade issues as soon as possible in the development cycle. ## How to test this PR locally We can run the integration tests using `AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^(TestFleetManagedUpgrade|TestStandaloneUpgrade)$" mage integration:test` or if you want to run just a specific version upgrade scenario we can use (for example only standalone upgrade from 8.9.0 to 8.10.0-SNAPSHOT): `AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade/Upgrade_8.9.0_to_8.10.0-SNAPSHOT$"` --------- Co-authored-by: Blake Rouse <[email protected]> (cherry picked from commit cfd059d) # Conflicts: # testing/integration/upgrade_test.go

…3173) This is an automatic backport of pull request #2955 done by [Mergify](https://mergify.com). --------- Co-authored-by: Paolo Chilà <[email protected]>

This PR filters out versions that too new from agent upgrade integration tests, allowing backport of those test on release branches. This PR needs to backported together with #2955 onto 8.9 branch We want to have working upgrade tests with relevant versions also on release branches

pchila added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog Testing labels Jun 28, 2023

pchila self-assigned this Jun 28, 2023

pchila requested a review from a team as a code owner June 28, 2023 08:56

pchila requested review from michalpristas and blakerouse June 28, 2023 08:56

mergify bot added the backport-skip label Jun 28, 2023

pchila requested a review from ycombinator June 29, 2023 06:34

ycombinator reviewed Jul 5, 2023

View reviewed changes

testing/integration/upgrade_test.go Outdated Show resolved Hide resolved

ycombinator reviewed Jul 5, 2023

View reviewed changes

testing/integration/upgrade_test.go Outdated Show resolved Hide resolved

ycombinator reviewed Jul 5, 2023

View reviewed changes

testing/integration/upgrade_test.go Outdated Show resolved Hide resolved

pchila force-pushed the hackathon/upgrade-integration-tests branch 2 times, most recently from de6ffd0 to 4ba5334 Compare July 13, 2023 13:01

pchila force-pushed the hackathon/upgrade-integration-tests branch from 4ba5334 to 2384643 Compare July 17, 2023 12:36

This was referenced Jul 20, 2023

[Integration Test] Upgrade failed default verification test #3101

Merged

Pass go test flags to integration tests #3109

Merged

pchila force-pushed the hackathon/upgrade-integration-tests branch from 2384643 to 608a4db Compare July 24, 2023 07:04

pchila force-pushed the hackathon/upgrade-integration-tests branch 2 times, most recently from 9d40f55 to ee5ae5e Compare July 25, 2023 07:01

pchila and others added 17 commits August 2, 2023 07:38

Introduce upgrade tests from released versions

7fe305b

use local packages for upgrade tests

c57b4eb

Skip signature verification in upgrade tests

0b9a849

refactor and cleanup upgrade tests

7bcc7d6

refactored version constraints in upgrade integration tests

11a9836

Use CLI to trigger upgrades in integration tests

a4feaea

add specific v1 proto health and version checks

d23c12d

Fix upgradeToSpecificSnapshot scenario

635ae90

Test upgrade for the current major and 2 most recent minors + latest …

bab55b8

…previous major version

fixup! Test upgrade for the current major and 2 most recent minors + …

da0c2fc

…latest previous major version

Fix InstallOpts for 7.x tests

2f82f21

fixup! Fix InstallOpts for 7.x tests

ccf9f45

Integrate PR feeback

08460e9

Update pkg/testing/fixture.go

56ca633

Co-authored-by: Blake Rouse <[email protected]>

Integrate PR feedback

24a12ce

fixup! Integrate PR feedback

b5962ed

fixup! fixup! Integrate PR feedback

bde7012

pchila force-pushed the hackathon/upgrade-integration-tests branch from 769022d to bde7012 Compare August 2, 2023 05:38

pchila merged commit cfd059d into elastic:main Aug 2, 2023
12 checks passed

pchila mentioned this pull request Aug 2, 2023

Skip versions newer than version under test for upgrade tests #3169

Merged

2 tasks

pchila added backport-v8.9.0 Automated backport with mergify and removed backport-skip labels Aug 3, 2023

mergify bot mentioned this pull request Aug 3, 2023

[8.9](backport #2955) Introduce upgrade tests from released versions #3173

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce upgrade tests from released versions #2955

Introduce upgrade tests from released versions #2955

pchila commented Jun 28, 2023 •

edited

Loading

elasticmachine commented Jun 28, 2023

mergify bot commented Jun 28, 2023

elasticmachine commented Jun 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 28, 2023 •

edited

Loading

mergify bot commented Jun 30, 2023

pchila commented Jul 13, 2023

pierrehilbert commented Jul 17, 2023

mergify bot commented Jul 17, 2023

pierrehilbert commented Jul 20, 2023

pchila commented Jul 24, 2023

pchila commented Aug 1, 2023

Introduce upgrade tests from released versions #2955

Introduce upgrade tests from released versions #2955

Conversation

pchila commented Jun 28, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Questions to ask yourself

elasticmachine commented Jun 28, 2023

mergify bot commented Jun 28, 2023

elasticmachine commented Jun 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jun 28, 2023 • edited Loading

🌐 Coverage report

mergify bot commented Jun 30, 2023

pchila commented Jul 13, 2023

pierrehilbert commented Jul 17, 2023

mergify bot commented Jul 17, 2023

pierrehilbert commented Jul 20, 2023

pchila commented Jul 24, 2023

pchila commented Aug 1, 2023

pchila commented Jun 28, 2023 •

edited

Loading

elasticmachine commented Jun 28, 2023 •

edited

Loading

elasticmachine commented Jun 28, 2023 •

edited

Loading