[Enhancement] Move knn lib building process to its standalone stage or pipeline

[peterzhuamazon]

Coming from #4379 #4687 and we are looking at possibilities to move knnlib compilation to its own stage or pipeline.

As of now, knnlib build is part of the knn build.gradle task, and will be run before the knn java plugin getting built.

Due to the uniqueness of this native lib, we could explore options to build it outside of the scope of core/plugin build.

As of now the knn team is suggesting building the lib, upload to S3, then download the lib during distribution build/assemble workflow.

There are also some limitations with the existing distribution build pipeline on Jenkins that needs to be taken care of.

Thanks.

[peterzhuamazon]

In short term:

• Trigger knnlib standalone pipeline once every release (ideally), upload to s3 with same version number as the release plugin version.
• Trigger distribution build with cron and build with the uploaded knnlib.
• If knnlib requires any updates just repeat first step

In long term:

• knnlib has its own version sit into its own repo. Every time when a release comes, maintainer cut a tag to trigger a build, upload to maven.
• Trigger distribution build with cron and build with knnlib pulled from maven.
• If knnlib requires any updates just repeat first step

[peterzhuamazon]

With the success of the AL2 compilation, we will go ahead with the S3 approach for now, and see if we can bring it up for 2.15.0. If not, we will fall back to use AL2 for all the builds. Else, we will use Almalinux8 for the build and AL2 for knnlib.

#4379 (comment)

Start checking the jenkins lib and write a POC for the knnlib pipeline.

[peterzhuamazon]

After discussion with team on the current progress, here are some of the points suggested by comments:

1. Since [Deprecation] Properly deprecate CentOS7 as CI build image/Supported OS and switch to Almalinux8 #4379 confirms that AL2 can compile the lib, we can fully deprecate CentOS7 by 2.15.0.
2. k-NN team should consider move the jni libs to its own repo, so they can independently version the lib outside of OpenSearch version, and release from the github actions.
3. We can use AL2 to compile core + plugins + knnlib for now, and switch to Almalinux8 in the next year. The existing pipeline is pretty stable for now.

Welcome comments,

Thanks.