Add step to publishPluginZipPublicationToMavenLocal in the build script

[cwperks]

Description

Adds a step to publishPluginZipPublicationToMavenLocal in the security plugin's build script. This will resolve an issue in the distribution build where flow framework is trying to get a "release" (non-snapshot) artifact of the security plugin at build time. See related issue here. The k-NN plugin runs this step in its build.sh script.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Maintenance

Issues Resolved

• [BUG] Flow-framework build failed because no opensearch-security dependency flow-framework#410

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

Moved this to draft. ./gradlew publishToMavenLocal is failing locally for me with the following error:

➜  security git:(publish-to-maven-local) ✗ ./gradlew publishToMavenLocal
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Mac OS X 14.2.1 (aarch64)
  JDK Version           : 11 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/cwperx/.sdkman/candidates/java/11.0.21-amzn
  Random Testing Seed   : 2C9F75A6312C4D23
  In FIPS 140 mode      : false
=======================================
> Task :generatePomFileForNebulaPublication FAILED

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':generatePomFileForNebulaPublication' (type 'GenerateMavenPom').
  - Gradle detected a problem with the following location: '/Users/cwperx/Projects/opensearch/security/build/distributions/opensearch-security-3.0.0.0-SNAPSHOT.pom'.

    Reason: Task ':publishMavenPublicationToMavenLocal' uses this output of task ':generatePomFileForNebulaPublication' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.

    Possible solutions:
      1. Declare task ':generatePomFileForNebulaPublication' as an input of ':publishMavenPublicationToMavenLocal'.
      2. Declare an explicit dependency on ':generatePomFileForNebulaPublication' from ':publishMavenPublicationToMavenLocal' using Task#dependsOn.
      3. Declare an explicit dependency on ':generatePomFileForNebulaPublication' from ':publishMavenPublicationToMavenLocal' using Task#mustRunAfter.

    For more information, please refer to https://docs.gradle.org/8.5/userguide/validation_problems.html#implicit_dependency in the Gradle documentation.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 2s
15 actionable tasks: 5 executed, 10 up-to-date

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (09051f4) 65.49% compared to head (b7e38e2) 65.50%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3959   +/-   ##
=======================================
  Coverage   65.49%   65.50%           
=======================================
  Files         298      298           
  Lines       21243    21246    +3     
  Branches     3460     3461    +1     
=======================================
+ Hits        13914    13918    +4     
+ Misses       5611     5608    -3     
- Partials     1718     1720    +2     

Files	Coverage Δ
...urity/ssl/transport/SecuritySSLNettyTransport.java	76.69% <100.00%> (+10.69%)	⬆️

... and 2 files with indirect coverage changes

[owaiskazi19]

@cwperks you can refer opensearch-project/flow-framework#226 to fix the above error.

[reta]

@cwperks @peternied we shouldn't publish JARs to Apache Maven, but the ZIP distribution only (which is done on the next line)

[cwperks]

@reta any idea why flow-framework cannot get the security zip if this line is in security's build.sh?

./gradlew publishPluginZipPublicationToZipStagingRepository -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER

Shouldn't the artifact be being published for other plugins to consume? This command works for me locally and publishes to <OPENSEARCH_SECURITY_ROOT>/build/local-staging-repo.

[peternied]

@cwperks @peternied we shouldn't publish JARs to Apache Maven, but the ZIP distribution only (which is done on the next line)

@reta Totally agreed; however, this change doesn't publish externally - I'm pretty sure you had the same reaction I did when I first saw this PR (I requested changes and then accepted). The build script is used when the distribution build is created and the command is publishToMavenLocal where the same machine in jenkins is used to build all of the artifacts in one place.

OpenSearch publishes to maven local in its scripts/build.sh

[reta]

The build script is used when the distribution build is created and the command is publishToMavenLocal where the same machine in jenkins is used to build all of the artifacts in one place.

Why should build script be published? It is owned by security plugin and only, but publishing it you would introduce a) build split brain (why only internal? what if someone accidentally publish it?) b) maintenance load for a team to maintain publishing artifacts that not needed

[cwperks]

I switched this to using publishPluginZipPublicationToMavenLocal, same as k-NN.

I'm trying to figure out how job-scheduler publishes the zip during distribution build time for dependent plugins.

[peternied]

Why should build script be published?

@reta This script is used by the distribution build process docs which operates differently than gradle builds. This problem was discovered when attempting to run the integration tests against a distribution build. I'm not if there is another kind of approach that could address this, do you have other ideas?

@zelinh Would you mind reviewing this PR and is there more context you can provide for the questions that @.reta has called out?

[reta]

@reta This script is used by the distribution build process docs which operates differently than gradle builds.

Hm ... I see it needs ZIP and with the ZIP publishing to Mavel Local, we should be all set:

 zipArchive group: 'org.opensearch.plugin', name:'opensearch-security', version: "${opensearch_build}" 

[cwperks]

@zelinh How does opensearch-build allow plugins to have a zipArchive dependency on job-scheduler? From what I am looking at in job-scheduler's build script it is not publishing the zip to maven local. There are lines in job-scheduler's build script to copy the zip to $OUTPUT/plugins.

I'm not sure if this is connected, but I see special handling for job-scheduler in opensearch-build. Is this how plugins are able to have a zipArchive dependency on job-scheduler (Example from AD)?

[cwperks]

Switched back to draft to prevent this from being merged. I'd like to understand how zipArchive dependency works for the integ tests in greater detail. Other plugins do not publish the zip to maven local even when other plugins have a zipArchive dependency on them (example from security-analytics which has a zipArchive dependency on alerting). This may not be necessary for the security plugin

[prudhvigodithi]

Let me explain this based on what I understood:

The zipArchive is a custom configuration that has been added in the configurations block. This configuration is used to group and declare dependencies related to OpenSearch plugins. It doesn't have a predefined meaning in Gradle rather, it's a configuration name chosen by the build script author.

configurations {
    zipArchive
}

By creating a custom configuration like zipArchive, the build script author can organize dependencies based on their purpose or use case. It's a way to group dependencies that share a common characteristic or belong to a specific category.

Now In the dependencies block, the OpenSearch plugin dependencies are associated with the zipArchive configuration:

Example as

dependencies {
    zipArchive group: 'org.opensearch.plugin', name:'alerting', version: "${opensearch_build}"
    zipArchive group: 'org.opensearch.plugin', name:'opensearch-notifications-core', version: "${opensearch_build}"
    zipArchive group: 'org.opensearch.plugin', name:'notifications', version: "${opensearch_build}"
    zipArchive group: 'org.opensearch.plugin', name:'opensearch-job-scheduler', version: "${opensearch_build}"
}

This means that these dependencies are associated with the zipArchive configuration when you later run a task that requires the zipArchive configuration (e.g compiling code that depends on these plugins), Gradle will only then download and include these dependencies in the classpath for that task. Just declaring under dependencies and not using zipArchive will not download those dependencies.

So @cwperks the Example from AD even though it has zipArchive group: 'org.opensearch.plugin', name:'opensearch-job-scheduler', version: "${opensearch_build}" It later never used the opensearch-job-scheduler.

Same with K-NN even though it has zipArchive group: 'org.opensearch.plugin', name:'opensearch-security', version: "${opensearch_build}" since it never uses zipArchive the 2.12.0 build passed for k-NN.

Now coming back to flow-framework and neural-search plugin it explicitly uses the zipArchive,
https://github.com/opensearch-project/neural-search/blob/2.x/build.gradle#L64-L83

configurations.zipArchive.asFileTree.each {
            if(it.name.contains("opensearch-security")){

Since security is not publishing to maven local gradle finds and fails the build as dependency not found with configuration zipArchive

A problem occurred evaluating root project 'opensearch-flow-framework'.
> Could not resolve all files for configuration ':zipArchive'.
   > Could not find org.opensearch.plugin:opensearch-security:2.12.0.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - file:/usr/share/opensearch/.m2/repository/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/plugin/opensearch-security/2.12.0.0/opensearch-security-2.12.0.0.pom
     Required by:
         project

This used to work for neural-search 2.11.0 release where then it does not have explicit zipArchive opensearch-security. This was added for 2.12.0 and the story should be the same for flow-framework.

Adding @gaiksaya @vibrantvarun @bbarani @jmazanec15 @owaiskazi19

[cwperks]

@prudhvigodithi Do you think it makes sense to add a step to publish to maven local for the security zip as part of security's build script? This PR should resolve Flow Framework's issue, but I'd like guidance from EE if this is the best way to fix their issue.