[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

reta · 2024-10-11T19:23:00Z

Is your feature request related to a problem? Please describe

At the moment, we do not publish plugins SNAPSHOT artifacts but min distribution only (see please https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/). However, it becomes increasingly demanded to have plugins SNAPSHOT artifacts published as well.

Describe the solution you'd like

Publish plugins SNAPSHOT artifacts under S3 bucket https://artifacts.opensearch.org/snapshots/plugins/ so OpenSearch tooling (opensearch-plugin, etc) could be used to install them for SNAPSHOT distributions.

Describe alternatives you've considered

Publish ZIPs to Apache Maven but that would change completely the distribution model for releases and snapshots.

Additional context

See please opensearch-project/OpenSearch#16219

The text was updated successfully, but these errors were encountered:

reta · 2024-10-11T19:23:08Z

CC @prudhvigodithi

prudhvigodithi · 2024-10-11T20:06:14Z

We have been pushing the created core-plugins already to https://ci.opensearch.org but not in https://artifacts.opensearch.org/snapshots/plugins/ format.

Example:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/rpm/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/deb/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

and we do the same for arm64

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thank you

reta · 2024-10-11T20:28:16Z

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thanks @prudhvigodithi . So this is somewhat inconsistent: all these 3 URLs point to the same plugin (at least they should) because the distributions of the plugins do not depend on the OpenSearch packaging. We could leverage that, indeed, but looks more like a workaround, not a solution.

peterzhuamazon · 2024-10-11T20:49:50Z

At the moment we just let every distribution build the native plugins zip, but when release we only release x64/tar version.

peterzhuamazon · 2024-10-11T20:50:11Z

So that we do not need to change the workflow too much.

dblock · 2024-10-23T15:59:04Z

Will this address my problem in opensearch-project/opensearch-api-specification#610?

I am using the 3.0 docker build and need to install one of the optional plugins.

Normal docker reference:

FROM opensearchstaging/opensearch:2.18.0@sha256:f311da085b968663fadf041bafc78b24dc21556f0726e625caf3560f7323a020

# attempt to install the plugin here
RUN ./bin/opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10380/linux/x64/tar/builds/opensearch/core-plugins/analysis-phonenumber-2.18.0.zip

The above example is problematic because (it doesn't work), but I also need to manually match the version to the docker image SHA somehow.

reta · 2024-10-23T16:03:20Z

Will this address my problem in opensearch-project/opensearch-api-specification#610?

It will @dblock

peterzhuamazon · 2024-10-24T16:17:16Z

After some talk on the triage meeting I fully understand what @reta is talking about.

Similar to:
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

Do a:
https://artifacts.opensearch.org/snapshots/plugins/opensearch/3.0.0-SNAPSHOT/<>.zip

We can start this with opensearch first, as I just checked we havent done any core snapshot for OSD core on artifacts.opensearch.org for a long time.

Thanks.

Divyaasm · 2024-10-28T22:31:08Z

Acceptance criteria:
Publish native plugins along side the core snapshot ->
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

peterzhuamazon · 2024-10-28T22:31:44Z

We also need to add the checksum files alongside the native plugin zips.
Probably need to use the same folder structure for snapshot as the release ones.

peterzhuamazon · 2024-10-28T23:30:29Z

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

reta · 2024-10-28T23:33:23Z

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

Thanks @peterzhuamazon, we only need to know the location, if it is different from the released versions fe

peterzhuamazon · 2024-10-30T15:24:51Z

Take a look in the next two days, assuming we should be able to add it as part of min snapshot workflows.

peterzhuamazon · 2024-10-30T19:32:57Z

After offline sync, we will use this folder structure:

https://artifacts.opensearch.org/snapshots/plugins/analysis-icu/2.10.0-SNAPSHOT/analysis-icu-2.10.0-SNAPSHOT.zip

peterzhuamazon · 2024-10-31T20:08:07Z

PRs:

Add core plugins snapshots publishing alongside min snapshots opensearch-build-libraries#556
Upload core plugins snapshots / gradle check updates in lib 7.3.2 #5165

peterzhuamazon · 2024-11-06T03:17:03Z

Hi @reta confirms 2.19.0-SNAPSHOT is up with core plugins
https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/1947/console

Ex: https://artifacts.opensearch.org/snapshots/plugins/analysis-icu/2.19.0-SNAPSHOT/analysis-icu-2.19.0-SNAPSHOT.zip

3.0.0-SNAPSHOT still in progress here: https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/1948/console
Should follow the same url pattern as above.

Thanks.

reta · 2024-11-06T14:19:39Z

Thanks @peterzhuamazon , we have one more issue to clarify, for official plugins we also sign them and publish signature, it seems like we don't do this for snapshots?

/opensearch-2.19.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-reactor-netty4                                                                                                                                                                                                     [15/178]
-> Installing transport-reactor-netty4                                                                                                                                     
-> Downloading transport-reactor-netty4 from opensearch                                                                                                                    
[=================================================] 100%                                                                                                                   
-> Failed installing transport-reactor-netty4                                                                                                                              
-> Rolling back transport-reactor-netty4                                                                                                                                   
-> Rolled back transport-reactor-netty4                                                                                                                                    
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/snapshots/plugins/transport-reactor-netty4/2.19.0-SNAPSHOT/transport-reactor-netty4-2.19.0-SNAPSHOT.zip.sig
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2013)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
        at java.base/java.net.URL.openStream(URL.java:1325)                                                                                                                
        at org.opensearch.plugins.InstallPluginCommand.urlOpenStream(InstallPluginCommand.java:490)
        at org.opensearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:630)
        at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:608)
        at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:316)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

peterzhuamazon · 2024-11-06T18:01:54Z

Thanks @peterzhuamazon , we have one more issue to clarify, for official plugins we also sign them and publish signature, it seems like we don't do this for snapshots?

/opensearch-2.19.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-reactor-netty4                                                                                                                                                                                                     [15/178]
-> Installing transport-reactor-netty4                                                                                                                                     
-> Downloading transport-reactor-netty4 from opensearch                                                                                                                    
[=================================================] 100%                                                                                                                   
-> Failed installing transport-reactor-netty4                                                                                                                              
-> Rolling back transport-reactor-netty4                                                                                                                                   
-> Rolled back transport-reactor-netty4                                                                                                                                    
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/snapshots/plugins/transport-reactor-netty4/2.19.0-SNAPSHOT/transport-reactor-netty4-2.19.0-SNAPSHOT.zip.sig
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2013)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
        at java.base/java.net.URL.openStream(URL.java:1325)                                                                                                                
        at org.opensearch.plugins.InstallPluginCommand.urlOpenStream(InstallPluginCommand.java:490)
        at org.opensearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:630)
        at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:608)
        at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:316)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

Yes we dont have signings for snapshots, on both plugins zips and min artifacts.
Do we need them in order to install?

peterzhuamazon · 2024-11-08T20:28:24Z

New run:

peterzhuamazon · 2024-11-08T21:00:16Z

Needs to add gnupg to macos for signature verifications.

+ curl -sSL https://artifacts.opensearch.org/publickeys/opensearch.pgp

+ gpg --import -

<>: line 1: gpg: command not found

script returned exit code 127

peterzhuamazon · 2024-11-08T21:01:22Z

More more PRs:

Adding gnupg to the macos agents for signature verifications opensearch-ci#503

peterzhuamazon · 2024-11-08T21:07:49Z

The above wont affect linux installation as that is already pushed.

peterzhuamazon · 2024-11-08T21:11:41Z

Weird I dont see the sig files for uploads.

reta · 2024-11-08T21:23:08Z

Yeah, me neither

peterzhuamazon · 2024-11-08T22:35:51Z

Out of no where the core-plugins location being null:

Running /var/jenkins/workspace/publish-opensearch-min-snapshots/src/run_sign.py null --sigtype .sig ...

peterzhuamazon · 2024-11-08T22:42:05Z

A re-run in test branch passed somehow, will just re-run production as I suspect just hiccups.

peterzhuamazon · 2024-11-08T22:45:08Z

Even worse, now the new macos agents cannot find pipenv, due to py39 not exist in macports?:


==> amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Error: Port py39-python-install not found
    amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Selecting 'python39' for 'python' failed: The specified group 'python' does not exist.
    amazon-ebs.Jenkins-Agent-MacOS13-ARM64-Mac2M2proMetal-13.6.7-2024-11-08T09-08-06Z: Selecting 'python39' for 'python3' failed: The specified group 'python3' does not exist.

peterzhuamazon · 2024-11-08T22:48:05Z

py39-python-install has been deleted as part of the macports:
https://ports.macports.org/port/py39-python-install/

peterzhuamazon · 2024-11-08T23:12:01Z

Even more PRs:

Replace py39-python-installer with python39 due to port removal opensearch-ci#504

peterzhuamazon · 2024-11-09T00:48:24Z

Seems like it is very random that var passed to signing will become null

Running /var/jenkins/workspace/publish-opensearch-min-snapshots/src/run_sign.py null --sigtype .sig ...

peterzhuamazon · 2024-11-09T00:50:27Z

On the side note windows takes forever to complete the signing.
I think we need to remove signing from normal artifacts now and only reserve for core plugins.

peterzhuamazon · 2024-11-09T03:58:28Z

Even Even more:

Only sign core plugins and ignore min artifacts opensearch-build-libraries#564

peterzhuamazon · 2024-11-11T23:22:19Z

One more PR:

Publish Min Artifacts with new 7.3.4 libs #5180

peterzhuamazon · 2024-11-12T17:25:13Z

I thinks this time it is publishing:

Hi @reta could you help confirm from your side on the installation? Thanks!

reta · 2024-11-12T19:16:10Z

Hi @reta could you help confirm from your side on the installation? Thanks!

Works like a charm, @peterzhuamazon !

$ /home/opensearch-3.0.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-nio
-> Installing transport-nio
-> Downloading transport-nio from opensearch
[=================================================] 100%   
-> Installed transport-nio with folder name transport-nio

$ ll plugins/transport-nio/
total 3.3M
-rw-r--r-- 1 user group  12K Nov 12 14:15 LICENSE.txt
-rw-r--r-- 1 user group 329K Nov 12 14:15 netty-buffer-4.1.114.Final.jar
-rw-r--r-- 1 user group 344K Nov 12 14:15 netty-codec-4.1.114.Final.jar
-rw-r--r-- 1 user group 653K Nov 12 14:15 netty-codec-http-4.1.114.Final.jar
-rw-r--r-- 1 user group 679K Nov 12 14:15 netty-common-4.1.114.Final.jar
-rw-r--r-- 1 user group 564K Nov 12 14:15 netty-handler-4.1.114.Final.jar
-rw-r--r-- 1 user group  37K Nov 12 14:15 netty-resolver-4.1.114.Final.jar
-rw-r--r-- 1 user group 510K Nov 12 14:15 netty-transport-4.1.114.Final.jar
-rw-r--r-- 1 user group  44K Nov 12 14:15 netty-transport-native-unix-common-4.1.114.Final.jar
-rw-r--r-- 1 user group  16K Nov 12 14:15 NOTICE.txt
-rw-r--r-- 1 user group  59K Nov 12 14:15 opensearch-nio-3.0.0-SNAPSHOT.jar
-rw-r--r-- 1 user group 1.9K Nov 12 14:15 plugin-descriptor.properties
-rw-r--r-- 1 user group 1.7K Nov 12 14:15 plugin-security.policy
-rw-r--r-- 1 user group  66K Nov 12 14:15 transport-nio-client-3.0.0-SNAPSHOT.jar

reta added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Oct 11, 2024

reta mentioned this issue Oct 22, 2024

Add pluginZip publication for plugins in the plugins folder opensearch-project/OpenSearch#16219

Closed

3 tasks

dblock mentioned this issue Oct 23, 2024

Add spec for opensearch phone plugin. opensearch-project/opensearch-api-specification#610

Closed

peterzhuamazon removed the untriaged Issues that have not yet been triaged label Oct 24, 2024

peterzhuamazon self-assigned this Oct 30, 2024

peterzhuamazon added jenkins Jenkins related issue Build Libraries & Interfaces labels Oct 30, 2024

peterzhuamazon mentioned this issue Oct 31, 2024

Add core plugins snapshots publishing alongside min snapshots opensearch-project/opensearch-build-libraries#556

Merged

peterzhuamazon mentioned this issue Nov 5, 2024

Upload core plugins snapshots / gradle check updates in lib 7.3.2 #5165

Merged

prudhvigodithi closed this as completed in #5165 Nov 6, 2024

reta mentioned this issue Nov 6, 2024

Support installing plugin SNAPSHOTs with SNASPHOT distribution opensearch-project/OpenSearch#16581

Open

3 tasks

peterzhuamazon reopened this Nov 8, 2024

github-actions bot added the untriaged Issues that have not yet been triaged label Nov 8, 2024

peterzhuamazon mentioned this issue Nov 8, 2024

Adding gnupg to the macos agents for signature verifications opensearch-project/opensearch-ci#503

Merged

peterzhuamazon removed the untriaged Issues that have not yet been triaged label Nov 8, 2024

peterzhuamazon closed this as completed in opensearch-project/opensearch-ci#503 Nov 8, 2024

peterzhuamazon reopened this Nov 8, 2024

github-actions bot added the untriaged Issues that have not yet been triaged label Nov 8, 2024

peterzhuamazon removed the untriaged Issues that have not yet been triaged label Nov 8, 2024

peterzhuamazon mentioned this issue Nov 8, 2024

Replace py39-python-installer with python39 due to port removal opensearch-project/opensearch-ci#504

Merged

peterzhuamazon mentioned this issue Nov 9, 2024

Only sign core plugins and ignore min artifacts opensearch-project/opensearch-build-libraries#564

Merged

This was referenced Nov 9, 2024

Update Jenkins with gnupg python39 macos agents opensearch-project/opensearch-ci#505

Merged

Publish Min Artifacts with new 7.3.4 libs #5180

Merged

peterzhuamazon closed this as completed in #5180 Nov 12, 2024

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

Comments

reta commented Oct 11, 2024

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

reta commented Oct 11, 2024

prudhvigodithi commented Oct 11, 2024 • edited Loading

reta commented Oct 11, 2024 • edited Loading

peterzhuamazon commented Oct 11, 2024

peterzhuamazon commented Oct 11, 2024

dblock commented Oct 23, 2024

reta commented Oct 23, 2024

peterzhuamazon commented Oct 24, 2024

Divyaasm commented Oct 28, 2024

peterzhuamazon commented Oct 28, 2024 • edited Loading

peterzhuamazon commented Oct 28, 2024 • edited Loading

reta commented Oct 28, 2024

peterzhuamazon commented Oct 30, 2024

peterzhuamazon commented Oct 30, 2024 • edited Loading

peterzhuamazon commented Oct 31, 2024 • edited Loading

peterzhuamazon commented Nov 6, 2024 • edited Loading

reta commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024 • edited Loading

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

reta commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 8, 2024

peterzhuamazon commented Nov 9, 2024

peterzhuamazon commented Nov 9, 2024

peterzhuamazon commented Nov 9, 2024

peterzhuamazon commented Nov 11, 2024

peterzhuamazon commented Nov 12, 2024

reta commented Nov 12, 2024 • edited Loading

prudhvigodithi commented Oct 11, 2024 •

edited

Loading

reta commented Oct 11, 2024 •

edited

Loading

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

peterzhuamazon commented Oct 30, 2024 •

edited

Loading

peterzhuamazon commented Oct 31, 2024 •

edited

Loading

peterzhuamazon commented Nov 6, 2024 •

edited

Loading

peterzhuamazon commented Nov 8, 2024 •

edited

Loading

reta commented Nov 12, 2024 •

edited

Loading