- Managing OpenSearch Plugins
We use meta to manage OpenSearch and OpenSearch Dashoards plugins as a set. There are two sets: OpenSearch Plugins and OpenSearch Dashboards Plugins. If you need a meta project for all the repositories in the (opensearch-project organization](https://github.com/opensearch-project/), see project-meta.
Install and configure GitHub CLI from https://cli.github.com/manual/. Authenticate with gh auth login and ensure that it works, e.g. gh issue list.
npm installSee package.json for all dependencies being installed.
meta git updateUse meta git pull to subsequently pull the latest revisions.
meta gh issue listmeta project import plugin [email protected]:opensearch-project/plugin.gitInstall ghi, e.g. brew install ghi.
meta exec "ghi label 'backwards-compatibility' -c '#773AA8'
This makes it easy to create version labels.
meta exec "ghi label 'untriaged' -c '#fbca04'"
meta exec "ghi label 'v1.0.0' -c '#d4c5f9'"
meta exec "ghi label 'v1.1.0' -c '#c5def5'"
meta exec "ghi label 'v1.2.0' -c '#bfdadc'"
meta exec "ghi label 'v2.0.0' -c '#b94c47'"
Alternate option if you are having issues with ghi:
meta exec "gh label create v2.2.0 -c '#537DB0'"
Create a file for the issue body, e.g. issue.md.
meta exec "gh issue create --label backwards-compatibility --title 'Ensure backwards compatibility with ODFE' --body-file ../issue.md"
In opensearch-build#497 we needed to remove integtest.sh from each repo.
We'll be pushing to forks. Make sure to replace username with your GitHub username below and to fork all the repos first.
meta exec "gh repo fork"
Ensure that a remote is setup for each plugin pointing to our forks.
meta exec "git remote get-url origin | sed s/opensearch-project/username/g | xargs git remote add username"
Remove a file, e.g. integtest.sh, commit and push.
meta exec "git rm integtest.sh"
meta exec "git add --all"
meta exec "git checkout -b remove-integtest-sh"
meta exec "git commit -s -m 'Removed integtest.sh.'"
meta exec "git push username remove-integtest-sh"
meta exec "gh pr create --title 'Removing default integtest.sh.' --body='Coming from https://github.com/opensearch-project/opensearch-build/issues/497, removing default integtest.sh.'"
Because one cannot install an older version of a plugin on a newer version of OpenSearch (see OpenSearch#1707), it's common to have to increment versions in all plugins, without making other changes. This was the case in the 1.2.3 release.
See also opensearch-build#1375 which aims to supersede this labor-intensive process with plugins auto-incrementing versions for the next development iteration.
The auto-increment workflow in OpenSearch#1816 will increment the version in OpenSearch patch branch, e.g. 1.2, OpenSearch#1758, and backport the version increment change in OpenSearch to 1.x (OpenSearch#1759) and main (OpenSearch#1760).
A workflow will create a new manifest that only contains OpenSearch, e.g. opensearch-build#1369. After this manifest is merged, wait for a successful SNAPSHOT build.
In the examples below we are incrementing 1.2.2 to 1.2.3.
export INCREMENT_BASE=1.2
export INCREMENT_FROM=1.2.2
export INCREMENT_TO=1.2.3
Check out and update the 1.2 branch.
meta git update
meta git pull origin
meta git checkout $INCREMENT_BASE
meta git pull origin $INCREMENT_BASE
Replace known versions.
find . -name build.gradle -print0 | xargs -0 sed -i "s/$INCREMENT_FROM-SNAPSHOT/$INCREMENT_TO-SNAPSHOT/g"
find . -wholename "*/.github/workflows/*.yml" -print0 | xargs -0 sed -i "s/$INCREMENT_FROM-SNAPSHOT/$INCREMENT_TO-SNAPSHOT/g"
find . -wholename "*/.github/workflows/*.yml" -print0 | xargs -0 sed -i "s/$INCREMENT_FROM.0-SNAPSHOT/$INCREMENT_TO.0-SNAPSHOT/g"
Plugins such as k-nn and security need some exact version replacements.
find . -name build.gradle -print0 | xargs -0 sed -i "s/$INCREMENT_FROM/$INCREMENT_TO/g"
find . -name CMakeLists.txt -print0 | xargs -0 sed -i "s/$INCREMENT_FROM.0/$INCREMENT_TO.0/g"
find . -name pom.xml -print0 | xargs -0 sed -i "s/$INCREMENT_FROM/$INCREMENT_TO/g"
find . -name plugin-descriptor.properties -print0 | xargs -0 sed -i "s/$INCREMENT_FROM/$INCREMENT_TO/g"
The cross-cluster-replication plugin needs an update in SecurityAdminWrapper.sh.
find . -name SecurityAdminWrapper.sh -print0 | xargs -0 sed -i "s/$INCREMENT_FROM/$INCREMENT_TO/g"
meta git checkout -b increment-to-$INCREMENT_TO
meta git add .
meta git commit -s -m "Incremented version to $INCREMENT_TO."
Create a remote for your own forks of the plugins. Replace <your-github-username>.
meta exec "gh repo fork --remote --remote-name <your-github-username>"
Ensure that a remote exists for your fork for every repo.
meta exec "git remote get-url origin | sed s/opensearch-project/<your-github-username>/g | xargs git remote add <your-github-username>"
Push the changes to your fork.
meta exec "git push <your-github-username> increment-to-$INCREMENT_TO"
An auto-increment workflow (see common-utils#106) will auto-increment the version, e.g. common-utils#105.
Make a pull request incrementing the version into job-scheduler that depends on OpenSearch, e.g. job-scheduler#110.
cd job-scheduler
gh pr create --title "Incremented version to $INCREMENT_TO." --body "Coming from https://github.com/opensearch-project/opensearch-build/issues/1365." --base $INCREMENT_BASE --label v$INCREMENT_TO'
Add common-utils and job-scheduler to the 1.2.3 manifest, e.g. opensearch-build#1374 and opensearch-build#1376 (you can combine both as job-scheduler doesn't depend on common-utils), and wait for a successful SNAPSHOT build.
Check that a snapshot build has been published, e.g. opensearch-min-1.2.3-SNAPSHOT-linux-x64-latest.tar.gz, which is required by most plugins' backwards compatibility tests. See opensearch-build#1261 for automating this.
Make a pull request incrementing the version into alerting, which contains notifications that index-management depends on, e.g. alerting#261. Add that plugin to the manifest and wait for a successful SNAPSHOT build, e.g. opensearch-build#1379.
Create pull requests referencing the parent release issue in opensearch-build. You will be prompted for where to push code, choose your fork.
meta exec 'gh pr create --title "Incremented version to $INCREMENT_TO." --body "Coming from https://github.com/opensearch-project/opensearch-build/issues/1365." --base $INCREMENT_BASE --label v$INCREMENT_TO'
In addition to the above changes, build and replace the job-scheduler SNAPSHOT jar in anomaly-detection, dashboards-reports, and index-management.
cd job-scheduler
git checkout $INCREMENT_BASE
git pull
./gradlew assemble
rm ../anomaly-detection/src/test/resources/job-scheduler/*
cp ./build/distributions/opensearch-job-scheduler-$INCREMENT_TO.0-SNAPSHOT.zip ../anomaly-detection/src/test/resources/job-scheduler/
rm -rf ../anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/$INCREMENT_FROM.0-SNAPSHOT
mkdir -p ../anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/$INCREMENT_TO.0-SNAPSHOT
cp ./build/distributions/opensearch-job-scheduler-$INCREMENT_TO.0-SNAPSHOT.zip ../anomaly-detection/src/test/resources/org/opensearch/ad/bwc/job-scheduler/$INCREMENT_TO.0-SNAPSHOT
rm ../dashboards-reports/reports-scheduler/src/test/resources/job-scheduler/*
cp ./build/distributions/opensearch-job-scheduler-$INCREMENT_TO.0-SNAPSHOT.zip ../dashboards-reports/reports-scheduler/src/test/resources/job-scheduler/
rm ../index-management/src/test/resources/job-scheduler/*
cp ./build/distributions/opensearch-job-scheduler-$INCREMENT_TO.0-SNAPSHOT.zip ../index-management/src/test/resources/job-scheduler/
For each of anomaly-detection, dashboards-reports, and index-management, use git add . to add the above updated-files, commit, and push an update, e.g. git push <your-github-username> increment-to-$INCREMENT_TO.
Ensure all plugins pass CI and the version increments have been merged. Add the remaining components to the manifest, e.g. opensearch-build#1380.