Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce upgrade tests from released versions #2955

Merged
merged 17 commits into from
Aug 2, 2023

Conversation

pchila
Copy link
Member

@pchila pchila commented Jun 28, 2023

What does this PR do?

This PR adds upgrade tests for both standalone and managed agent from some versions of the current and previous major release to the latest snapshot.

The tests will select:

  • up to 2 versions with the same major version as the current agent
  • up to 1 version from a previous major version
    These versions are retrieved using the Artifact API and working backwards through the versions, excluding snapshots.
    The up to part is because under certain circumstances (new major release, for example) we may not find enough versions fitting the criteria.

The tests TestStandaloneUpgrade and TestFleetManagedUpgrade will start a subtest (using t.Run()) for each version to be used as a starting point for the upgrade and then upgrade it to the latest version of agent.

Current limitations:

  • We are currently using the ArtifactFetcher so the starting elastic agent version all come from the artifact API. For all agent versions that allow the --skip-verify we use the packaged elastic agent as artifact for the upgrade via --sourceURI parameter

Why is it important?

We need to have automated tests for the common upgrade scenarios to detect upgrade issues as soon as possible in the development cycle.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Author's Checklist

  • [ ]

How to test this PR locally

We can run the integration tests using
AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^(TestFleetManagedUpgrade|TestStandaloneUpgrade)$" mage integration:test
or if you want to run just a specific version upgrade scenario we can use (for example only standalone upgrade from 8.9.0 to 8.10.0-SNAPSHOT):
AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade/Upgrade_8.9.0_to_8.10.0-SNAPSHOT$"

Related issues

Use cases

Screenshots

Logs

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

@pchila pchila added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog Testing labels Jun 28, 2023
@pchila pchila self-assigned this Jun 28, 2023
@pchila pchila requested a review from a team as a code owner June 28, 2023 08:56
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2023

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@elasticmachine
Copy link
Contributor

elasticmachine commented Jun 28, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-08-02T05:38:39.111+0000

  • Duration: 26 min 54 sec

Test stats 🧪

Test Results
Failed 0
Passed 6153
Skipped 31
Total 6184

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages.

  • run integration tests : Run the Elastic Agent Integration tests.

  • run end-to-end tests : Generate the packages and run the E2E Tests.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Contributor

elasticmachine commented Jun 28, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 98.718% (77/78) 👍
Files 68.498% (187/273) 👍
Classes 67.258% (341/507) 👍
Methods 53.796% (1070/1989) 👎 -0.027
Lines 40.034% (12250/30599) 👎 -0.006
Conditionals 100.0% (0/0) 💚

@pchila pchila requested a review from ycombinator June 29, 2023 06:34
@mergify
Copy link
Contributor

mergify bot commented Jun 30, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hackathon/upgrade-integration-tests upstream/hackathon/upgrade-integration-tests
git merge upstream/main
git push upstream hackathon/upgrade-integration-tests

@pchila pchila force-pushed the hackathon/upgrade-integration-tests branch 2 times, most recently from de6ffd0 to 4ba5334 Compare July 13, 2023 13:01
@pchila
Copy link
Member Author

pchila commented Jul 13, 2023

buildkite test this

1 similar comment
@pierrehilbert
Copy link
Contributor

buildkite test this

@mergify
Copy link
Contributor

mergify bot commented Jul 17, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hackathon/upgrade-integration-tests upstream/hackathon/upgrade-integration-tests
git merge upstream/main
git push upstream hackathon/upgrade-integration-tests

@pchila pchila force-pushed the hackathon/upgrade-integration-tests branch from 4ba5334 to 2384643 Compare July 17, 2023 12:36
@pierrehilbert
Copy link
Contributor

buildkite test this

@pchila
Copy link
Member Author

pchila commented Jul 24, 2023

Last integration test didn't run any tests... There's probably an issue with the test enumeration...

@pchila pchila force-pushed the hackathon/upgrade-integration-tests branch 2 times, most recently from 9d40f55 to ee5ae5e Compare July 25, 2023 07:01
@pchila
Copy link
Member Author

pchila commented Aug 1, 2023

buildkite test this

@pchila pchila force-pushed the hackathon/upgrade-integration-tests branch from 769022d to bde7012 Compare August 2, 2023 05:38
@pchila pchila merged commit cfd059d into elastic:main Aug 2, 2023
12 checks passed
pchila added a commit that referenced this pull request Aug 3, 2023
## What does this PR do?

This PR filters out versions that too new from agent upgrade integration
tests, allowing backport of those test on release branches.

This PR needs to backported together with #2955 onto 8.9 branch

## Why is it important?

We want to have working upgrade tests with relevant versions also on
release branches
@pchila pchila added backport-v8.9.0 Automated backport with mergify and removed backport-skip labels Aug 3, 2023
mergify bot pushed a commit that referenced this pull request Aug 3, 2023
## What does this PR do?

This PR filters out versions that too new from agent upgrade integration
tests, allowing backport of those test on release branches.

This PR needs to backported together with #2955 onto 8.9 branch

## Why is it important?

We want to have working upgrade tests with relevant versions also on
release branches

(cherry picked from commit 3de585e)

# Conflicts:
#	testing/integration/upgrade_test.go
mergify bot pushed a commit that referenced this pull request Aug 3, 2023
## What does this PR do?

This PR adds upgrade tests for both standalone and managed agent from
some versions of the current and previous major release to the latest
snapshot.

The tests will select:
- up to 2 versions with the same major version as the current agent
- up to 1 version from a previous major version
These versions are retrieved using the Artifact API and working
backwards through the versions, excluding snapshots.
The _up to_ part is because under certain circumstances (new major
release, for example) we may not find enough versions fitting the
criteria.

The tests `TestStandaloneUpgrade` and `TestFleetManagedUpgrade` will
start a subtest (using `t.Run()`) for each version to be used as a
starting point for the upgrade and then upgrade it to the latest version
of agent.

Current limitations:
- We are currently using the ArtifactFetcher so the starting elastic
agent version all come from the artifact API. For all agent versions
that allow the `--skip-verify` we use the packaged elastic agent as
artifact for the upgrade via `--sourceURI` parameter

## Why is it important?
We need to have automated tests for the common upgrade scenarios to
detect upgrade issues as soon as possible in the development cycle.

## How to test this PR locally

We can run the integration tests using
`AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run
^(TestFleetManagedUpgrade|TestStandaloneUpgrade)$" mage
integration:test`
or if you want to run just a specific version upgrade scenario we can
use (for example only standalone upgrade from 8.9.0 to 8.10.0-SNAPSHOT):
`AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run
^TestStandaloneUpgrade/Upgrade_8.9.0_to_8.10.0-SNAPSHOT$"`

---------

Co-authored-by: Blake Rouse <[email protected]>
(cherry picked from commit cfd059d)

# Conflicts:
#	testing/integration/upgrade_test.go
pchila added a commit that referenced this pull request Aug 3, 2023
## What does this PR do?

This PR adds upgrade tests for both standalone and managed agent from
some versions of the current and previous major release to the latest
snapshot.

The tests will select:
- up to 2 versions with the same major version as the current agent
- up to 1 version from a previous major version
These versions are retrieved using the Artifact API and working
backwards through the versions, excluding snapshots.
The _up to_ part is because under certain circumstances (new major
release, for example) we may not find enough versions fitting the
criteria.

The tests `TestStandaloneUpgrade` and `TestFleetManagedUpgrade` will
start a subtest (using `t.Run()`) for each version to be used as a
starting point for the upgrade and then upgrade it to the latest version
of agent.

Current limitations:
- We are currently using the ArtifactFetcher so the starting elastic
agent version all come from the artifact API. For all agent versions
that allow the `--skip-verify` we use the packaged elastic agent as
artifact for the upgrade via `--sourceURI` parameter

## Why is it important?
We need to have automated tests for the common upgrade scenarios to
detect upgrade issues as soon as possible in the development cycle.

## How to test this PR locally

We can run the integration tests using
`AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run
^(TestFleetManagedUpgrade|TestStandaloneUpgrade)$" mage
integration:test`
or if you want to run just a specific version upgrade scenario we can
use (for example only standalone upgrade from 8.9.0 to 8.10.0-SNAPSHOT):
`AGENT_VERSION="8.10.0-SNAPSHOT" GOTEST_FLAGS="-test.run
^TestStandaloneUpgrade/Upgrade_8.9.0_to_8.10.0-SNAPSHOT$"`

---------

Co-authored-by: Blake Rouse <[email protected]>
(cherry picked from commit cfd059d)

# Conflicts:
#	testing/integration/upgrade_test.go
pchila added a commit that referenced this pull request Aug 4, 2023
…3173)

This is an automatic backport of pull request #2955 done by
[Mergify](https://mergify.com).
---------

Co-authored-by: Paolo Chilà <[email protected]>
pchila added a commit that referenced this pull request Aug 4, 2023
This PR filters out versions that too new from agent upgrade integration
tests, allowing backport of those test on release branches.

This PR needs to backported together with #2955 onto 8.9 branch

We want to have working upgrade tests with relevant versions also on
release branches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.9.0 Automated backport with mergify skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants