[RELEASE] Release version 2.9.0

[github-actions]

Release OpenSearch and OpenSearch Dashboards 2.9.0

I noticed that a manifest was automatically created in manifests/2.9.0. Please follow the following checklist to make a release.

How to use this issue

This Release Issue

This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.

Release Steps

There are several steps to the release process, these steps are completed as the whole release and components that are behind present risk to the release. The release owner completes the tasks in this ticket, whereas component owners resolve tasks on their ticket in their repositories.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.

Component List

To aid in understanding the state of the release there is a table with status indicating each component state. This is updated based on the status of the component issues.

Preparation

• Assign this issue to a release owner.
• Declare a pencils down date for new features to be merged.
• REPLACE_RELEASE-minus-14-days is pencils down date for feature freeze.
• Update the Campaigns section to include monitoring campaigns during this release.
• Update this issue so all __REPLACE_RELEASE-__ placeholders have actual dates.
• Document any new quality requirements or changes.
• Finalize scope and feature set and update the Public Roadmap.
• Create a release issue in every component repo based on component release issue template and link back to this issue, update Components section with these links.
• Ensure the label is created in each component repo for this new version, and the next minor release. Create a version label
• Ensure that all release issues created above are assigned to an owner in the component team.
• Increase the build frequency for the this release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in jenkinsFile.

Campaigns

REPLACE with OpenSearch wide initiatives to improve quality and consistency.

Release Branch - July 10 2023

• Component versions are auto-incremented to 2.9.0 version.
• Plugins team to ensure working CI in component repositories and merge the version increment PRs.
• OpenSearch / OpenSearch-Dashboards core cut branch <MajorVersion>.<MinorVersion> early.
• All component repos create <MajorVersion>.<MinorVersion> branch for the release.

Feature Freeze - Ends - July 11 2023

• OpenSearch / OpenSearch-Dashboards core and components teams finalize their features.
• Create Jenkins workflows that run daily snapshot builds for OpenSearch and OpenSearch Dashboards.
• Make pull requests to add each component to manifests/2.9.0/opensearch-2.9.0.yml and manifests/2.9.0/opensearch-dashboards-2.9.0.yml with the corresponding checks.

Code Complete - _Ends _ - July 11 2023

• Code Complete: Make sure that the code for this specific version of the release is ready and the branch corresponding to this release has been added to this release version manifest.
• Verify pull requests to add each component to manifests/2.9.0/opensearch-2.9.0.yml and manifests/2.9.0/opensearch-dashboards-2.9.0.yml have been merged.
• Gather, review and combine the release notes from components repositories.

Release testing - July 12 2023 - July 14 2023 End of July 18, 2023

• Declare a release candidate build, and provide the instructions with the release candidates for teams on testing
• Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary else manually run the build workflow and declare new release candidate.
• After generate the release candidates, raise PR to lock input manifest refs of both OS and OSD builds with the commit ids from distribution manifest of the release candidates.
• Sanity Testing : Sanity testing and fixing of critical issues found by teams. Teams test their components within the distribution, ensuring integration, backwards compatibility, and perf tests pass.
• Publish all test results in the comments of this issue.

Performance testing validation - July 13 2023 - July 14 2023 End of July 19, 2023

• Performance tests do not show a regression.
• Longevity tests do not show any issues.

Release - July 18 2023 End of July 20, 2023 July 24th 2023 at 11.30 am PT

• Verify all issues labeled v2.9.0 in all projects have been resolved.
• Complete documentation for this release.
• Author blog post for this release.
• __REPLACE_RELEASE-minus-1-day - Publish this release on opensearch.org.
• __REPLACE_RELEASE-day - Publish a blog post - release is launched!

Post Release - July 18 2023 End of July 20, 2023 End of July 24th 2023

• Create release tags for each component (Jenkins job name: release-tag-creation).
• Replace refs in manifests/2.9.0 with tags and remove checks.
• If this is a major or minor version release, stop building previous patch version.
• Generate distribution release notes reviewed by PM team for opensearch-build repository.
• Increment version for Helm Charts (sample PR) for the 2.9.0 release.
• Increment version for Ansible Charts (sample PR) for the 2.9.0 release.
• Prepare for next patch release by incrementing patch versions for each component.
• Update this template with any new or missed steps.
• Create an issue for a retrospective, solicit feedback, and publish a summary.

Documentation Issues

Components

OpenSearch

Component	Issues tagged with 2.9.0
opensearch-project/OpenSearch#8417
opensearch-project/OpenSearch-Dashboards#4486
opensearch-project/common-utils#467
opensearch-project/k-NN#944
opensearch-project/geospatial#344
opensearch-project/security#2907
opensearch-project/cross-cluster-replication#1005
opensearch-project/ml-commons#1020
opensearch-project/neural-search#203
opensearch-project/notifications#695
opensearch-project/observability#1558
opensearch-project/reporting#717
opensearch-project/sql#1786
opensearch-project/asynchronous-search#299
opensearch-project/anomaly-detection#934
opensearch-project/alerting#972
opensearch-project/security-analytics#471
opensearch-project/index-management#826
opensearch-project/performance-analyzer#488

OpenSearch Dashboards

Component	Issues tagged with 2.9.0
opensearch-project/OpenSearch-Dashboards#4486
opensearch-project/opensearch-dashboards-functional-test#732
opensearch-project/dashboards-observability#579
opensearch-project/dashboards-reporting#130
opensearch-project/dashboards-visualizations#193
opensearch-project/dashboards-query-workbench#86
opensearch-project/dashboards-maps#427
opensearch-project/anomaly-detection-dashboards-plugin#517
opensearch-project/ml-commons-dashboards#223
opensearch-project/index-management-dashboards-plugin#796
opensearch-project/dashboards-notifications#62
opensearch-project/alerting-dashboards-plugin#584
opensearch-project/security-analytics-dashboards-plugin#637
opensearch-project/security-dashboards-plugin#1489
opensearch-project/dashboards-search-relevance#225

Legend

Symbol	Meaning
🟢	On track with overall release
🟡	Missed last milestone
🔴	Missed multiple milestones

[peterzhuamazon]

Status Updates:

Campaigns:

• Remove AngularJS OSD: @AMoo-Miki @ananzh @abbyhu2000
  • [Initiative] Remove AngularJS as dependency OpenSearch-Dashboards#1560 (comment)
  • Dashboard De-Angular Design OpenSearch-Dashboards#4317
• Validate plugins compatibility with segment replication [Meta] Validate plugins compatibility with segment replication OpenSearch#8211

Issues:

• Cmake upgrade Upgrade CMake to version: 3.23.3 #3706
• Release Label creation automation #3661
• ML BE/FE GA: @ylwu-amzn @ruanyl @wanglam
  • Example PRs in 2.8.0 that got reverted later:
    • feat: enable ml-commons-dashboards plugin by default ml-commons-dashboards#200
    • feat: remove ml-commons-dashboards experiment keyword in comments #3578
  • New 2.9.0 PRs:
    • Remove ml-commons-dashboards experiment keyword in config file #3705
    • Revert "Revert remove experiment banner and enable by default (#203)" ml-commons-dashboards#229
• ML IntegTest with new openai/cohere test:
  • Add integration tests for the remote inference feature ml-commons#1073
• KNN issues:
  • [AUTOCUT] Distribution Build Failed for k-NN-2.9.0 (Unittest error on LINUX and FAISS compilation on Windows) k-NN#975
• Docker Copy Issues:
  • [Enhancement] Add authentication to docker copy just like in gradle check #3737
    • Add docker login for both source and dest registries opensearch-build-libraries#249
    • Update docker related copy lib to new version #3800

PRs:

• Add osd 2.9.0 related manifests to build repo #3659
• [Vis Augmenter] Fix stats API visualization ID bug OpenSearch-Dashboards#4565 
 
• fix 2.9 ci observability#1569
• 
https://github.com/opensearch-project/OpenSearch/pull/8460

• [Backport 2.9] Fix get alerts api to also return chained alerts with default params. and skip audit state alerts alerting#1023
• h
ttps://github.com/Fix get alerts api to also return chained alerts with default params. and skip audit state alerts alerting#1020
 
• Fixing the k-nn build for windows platform after upgrading Faiss k-NN#980
• 
https://github.com/opensearch-project/alerting-dashboards-plugin/pull/634

• [Backport 2.9] Fix line to vega conversion bug OpenSearch-Dashboards#4558
• 
Fix line to vega conversion bug OpenSearch-Dashboards#4554
• [Backport 2.9] [Vis Augmenter] Fix stats API visualization ID bug OpenSearch-Dashboards#4567
• [Backport 2.9] Add shell bash settings for windows github actions to export credentials ml-commons#1120
• 
Add shell bash settings for windows github actions to export credentials ml-commons#1116
• [Backport 2.9] Increase sleep time on remote inference integration tests ml-commons#1125
• [Backport 2.9] fix encoding US-ASCII error ml-commons#1128
• revert integrations change metrics dashboards-observability#672
• [Backport 2.9] add workflow type field exists check in search monitors action to return both workflows and monitors on search alerting#1027
• Only check the gcc version ceiling when it is on linux k-NN#991
• Update the gcc version to 7 for knn lib on arm64 #3735
• Upgrade gcc to 8 with devtoolset-8 for k-NN faiss lib compilation #3732

[prudhvigodithi]

Thanks @peterzhuamazon I will keep editing your comment as we go through the release.

[peterzhuamazon]

ML Commons Undeploy bug on Windows:

• [BUG] Unable to redeploy a model once undeployed on Windows ml-commons#1149

[prudhvigodithi]

We have a PR out for OSD that we will backport to 2.9 and is needed for the 2.9 release.
Additionally, we are going to have another fix in related to this for Alerting Dashboards plugin and Anomaly Detection Dashboards plugin. These fixes are very isolated and would not affect other plugins.

New OSD RC 6391 generated on 07/20/2023, the OS RC is still the same as the build 8184

OSD: https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch-dashboards/detail/distribution-build-opensearch-dashboards/6391/

Integ:

• arm64: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/3696/pipeline
• x64: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/3695/pipeline
  rpm:
• arm64: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/3697/pipeline
• x64: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/3698/pipeline
  docker:
• Build: https://build.ci.opensearch.org/blue/organizations/jenkins/docker-build/detail/docker-build/3576/pipeline/
• Scan: https://build.ci.opensearch.org/job/docker-scan/1698/artifact/scan_docker_image.txt

Tar arm64

2023-07-20 17:09:16 INFO     | OpenSearch-Dashboards | without-security     | PASS  |
2023-07-20 17:43:53 ERROR    | alertingDashboards   | with-security        | FAIL  |
2023-07-20 17:43:53 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-07-20 16:46:56 ERROR    | queryWorkbenchDashboards | with-security        | FAIL  |
2023-07-20 16:46:56 ERROR    | queryWorkbenchDashboards | without-security     | FAIL  |
2023-07-20 17:48:03 ERROR    | securityAnalyticsDashboards | with-security        | FAIL  |
2023-07-20 17:48:03 ERROR    | securityAnalyticsDashboards | without-security     | FAIL  |

Tar x64

2023-07-20 17:23:01 ERROR    | alertingDashboards   | with-security        | FAIL  |
2023-07-20 17:23:01 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-07-20 17:03:20 ERROR    | observabilityDashboards | with-security        | FAIL  |
2023-07-20 17:03:20 ERROR    | observabilityDashboards | without-security     | FAIL  |
2023-07-20 16:42:49 ERROR    | queryWorkbenchDashboards | with-security        | FAIL  |
2023-07-20 16:42:49 ERROR    | queryWorkbenchDashboards | without-security     | FAIL  |
2023-07-20 17:00:35 ERROR    | reportsDashboards    | without-security     | FAIL  |
2023-07-20 17:32:26 ERROR    | securityAnalyticsDashboards | with-security        | FAIL  |
2023-07-20 17:32:26 ERROR    | securityAnalyticsDashboards | without-security     | FAIL  |

RPM arm64:

2023-07-20 18:22:32 ERROR    | OpenSearch-Dashboards | with-security        | FAIL  |
2023-07-20 18:56:57 ERROR    | alertingDashboards   | with-security        | FAIL  |
2023-07-20 18:56:57 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-07-20 18:09:19 ERROR    | notificationsDashboards | without-security     | FAIL  |
2023-07-20 18:23:04 ERROR    | observabilityDashboards | with-security        | FAIL  |
2023-07-20 17:59:56 ERROR    | queryWorkbenchDashboards | with-security        | FAIL  |
2023-07-20 17:59:56 ERROR    | queryWorkbenchDashboards | without-security     | FAIL  |
2023-07-20 18:59:00 ERROR    | securityAnalyticsDashboards | with-security        | FAIL  |
2023-07-20 18:59:00 ERROR    | securityAnalyticsDashboards | without-security     | FAIL  |

RPM x64:

2023-07-20 18:36:58 ERROR    | alertingDashboards   | with-security        | FAIL  |
2023-07-20 18:36:58 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-07-20 18:04:19 ERROR    | customImportMapDashboards | with-security        | FAIL  |
2023-07-20 18:05:49 ERROR    | notificationsDashboards | without-security     | FAIL  |
2023-07-20 17:58:35 ERROR    | queryWorkbenchDashboards | with-security        | FAIL  |
2023-07-20 17:58:35 ERROR    | queryWorkbenchDashboards | without-security     | FAIL  |
2023-07-20 18:16:36 ERROR    | reportsDashboards    | without-security     | FAIL  |
2023-07-20 18:45:59 ERROR    | securityAnalyticsDashboards | with-security        | FAIL  |
2023-07-20 18:45:59 ERROR    | securityAnalyticsDashboards | without-security     | FAIL  |

[peterzhuamazon]

Windows OSD Test failures on RC3:

alertingDashboards

[lezzago]

Signing off alertingDashboards and securityAnalyticsDashboards to green.

[prudhvigodithi]

Integ test results #3616 (comment) manual sign off for 3nd OSD RC Generated on 07/20/2023

Repo	Integ-Tests RC2
OpenSearch-Dashboards	🟢
alertingDashboards	🟢
queryWorkbenchDashboards	🟢
securityAnalyticsDashboards	🟢
reportsDashboards	🟢
observabilityDashboards	🟢
notificationsDashboards	🟢
customImportMapDashboards	🟢

[ohltyler]

OSD signed off - video shows success, same issue as seen in previous RC builds

[xluo-aws]

dashboardsnotification signed off - same problem as previous RC builds due to historical data not clean up.

[heemin32]

customImportMapDashboards signed off - The recorded video showed the test actually succeeded.

[kavithacm]

Signing off on observability, reports dashboards and query workbench - same issue as previous RC build

[prudhvigodithi]

OpenSearch 2.9.0 Release Checklist

Checklist for the release scheduled on July 24th 2023 at 11.30 am PT.

Pre-Release activities

• Promote Repos.

  • • OS
      https://build.ci.opensearch.org/job/distribution-promote-repos/58/
      https://build.ci.opensearch.org/job/distribution-promote-repos/59/
  • • OSD
      https://build.ci.opensearch.org/job/distribution-promote-repos/60/
      https://build.ci.opensearch.org/job/distribution-promote-repos/61/

• Promote artifacts.

  • • Windows
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/280/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/281/
  • • Linux Debian
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/282/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/283/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/284/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/285/
  • • Linux RPM
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/286/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/287/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/288/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/289/
  • • Linux TAR
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/290/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/291/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/292/
      https://build.ci.opensearch.org/job/distribution-promote-artifacts/293/

• Consolidated Release Notes: Open PR: [Release 2.9.0] Consolidated release notes #3803.

Release activities

• Docker Promotion. https://build.ci.opensearch.org/blue/organizations/jenkins/docker-promotion/detail/docker-promotion/36/pipeline/
• Release Validation part 1.
  • • OpenSearch and OpenSearch Dashboard Validation
  • • Validate the native plugin installation. Sample Validation: [RELEASE] Release version 2.9.0 #3616 (comment)
• Merge consolidated release notes PR.
• Website and Documentation Changes.
  • • Merge staging website PR.
  • • Promote the website changes to prod.
  • • Add website alert. https://github.com/opensearch-project/project-website/pull/1817/files
• Release Validation part 2.
  • • Validate the artifact download URL's and signatures.
      gpg: Good signature from "OpenSearch project <[email protected]>"
• Release Validation part 3.
  • • Trigger the validation build https://build.ci.opensearch.org/job/distribution-validation/ (Search for Completed validation for <> in the logs)
      https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-validation/detail/distribution-validation/4/pipeline
• Maven Promotion. https://build.ci.opensearch.org/blue/organizations/jenkins/publish-to-maven/detail/publish-to-maven/20/pipeline
  https://repo1.maven.org/maven2/org/opensearch/opensearch/2.9.0/
  Sample Validation: mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://repo1.maven.org/maven2 -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.9.0.0:zip
• Publish blog posts. https://www.opensearch.org/blog/introducing-opensearch-2.9.0/
• Advertise on Social Media. https://www.linkedin.com/posts/opensearch-project_opensearch-activity-7089402074509053952-pYEI?utm_source=share&utm_medium=member_desktop
• Post on public slack and Github Release issue.

Post-Release activities

• Release Tags.
  https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-release-tag-creation/detail/distribution-release-tag-creation/83/pipeline
  https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-release-tag-creation/detail/distribution-release-tag-creation/84/pipeline
• Input Manifest Update. [Release 2.9.0] Input Manifest tag update #3807
• Decrease the build frequency.
• OpenSearch Build Release notes. https://github.com/opensearch-project/opensearch-build/releases/tag/2.9.0
• Retrospective Issue. [Retrospective] Release 2.9.0 #3808
• Helm and Ansible Playbook release.
  [Release 2.9.0] Chart upgrade helm-charts#453
  Multi node Installation ansible-playbook#13
• Close the created [AUTOCUT] issues
  Distribution Build [AUTOCUT] issues
  Integration tests [AUTOCUT] issues
• Upcoming Release Preparation.
  At this point the upcoming release preparation, the following items are fully automated.
  • • Upcoming Release OpenSearch Version Increment.
  • • Upcoming Release OpenSearch Dashboards Version Increment.
  • • Upcoming Release Label Creation.
  • • Upcoming Build Repo Release Issue Creation.
  • • Upcoming Components Release Issue Creation.

[prudhvigodithi]

Native plugin validation tested on apt.

ubuntu@ip-172-31-89-54:/usr/share/opensearch/bin$ sudo ./opensearch-plugin install repository-s3 --batch
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission config#plus read
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission aws.configFile read,write
* java.util.PropertyPermission aws.sharedCredentialsFile read,write
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
* java.util.PropertyPermission opensearch.path.conf read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-s3 with folder name repository-s3

[prudhvigodithi]

Closing this issue as the release 2.9.0 is out, please ass your thoughts/feedback on the retro issue #3808.
Thank you