[PROPOSAL] Reduce GitHub Action jobs against any OpenSearch version

[shyim]

What/Why

What are you proposing?

Currently, we test in the Clients against almost any existing OpenSearch version and even for unreleased versions by compiling OpenSearch own in the Job.

The API requests between patch/minor version should not differ really much, and the pipeline shows the same error on all of them. The tests inside the OpenSearch main repository should test that the API works basically in each situation as the clients just crafts that API requests.

The Unreleased OpenSearch Tests takes ~17 minutes per job just to startup because they have to compile Java first and that is really waste of time and resources. We don't pay at GitHub anything for resources, but at the end it's still computing power = co2.

Example from PHP Client:

• Removing Unreleased versions would save at least ~68 minutes per commit
• The matrix for normal is about 60 jobs (15 versions * 4 php versions) each job is running a minute. We could save here also several minutes

What problems are you trying to solve?

• Make the CI faster
• Think more about the resources we use and save energy (co2)

What is the developer experience going to be?

As the OpenSearch main repository tests enough, the API we have to just make sure that the client works in that language

Are there any security considerations?

Nope

Are there any breaking changes to the API

Nope

What is the user experience going to be?

Nothing

[dblock]

I think catching the unreleased version errors is very valuable, but I would reduce the matrix to the latest 1.x and 2.x.

[shyim]

The unreleased ones takes the longest time. But the ci of the server repository should take care that the API works basically

[wbeckler]

What if the unreleased version were tested on a different cadence, independent of client builds? It doesn't make sense for a client build to succeed, and then an hour later to fail, just because of a change in OpenSearch core. It throws off the momentum of the client developer. And if we do want to test off core, maybe it makes more sense to test off nightly builds rather than having to compile it. Thoughts?

[dblock]

The standard (TM) way to do this is to ignore unreleased build failures. I also really like @wbeckler's suggestion of running those tests on a cron, and auto-opening an issue/adding to an existing one saying: client is currently not working against the next version, so clients can anticipate.

[wbeckler]

@shyim What would be the approximate impact on build time if you adopted all of the above mentioned efficiencies, including moving the unreleased build tests to a different mechanism?

[VachaShah]

The main idea towards adding tests against unreleased versions was so that we can catch changes on the server that cause breaking effects on the client. This helped us in the development of 2.0 clients since we didn't have to wait for the 2.0 server version to be released. Having said that, I really like the idea of moving the unreleased tests to a cron schedule and create issues if they fail. This will also help in better narrowing down the commits that cause breaking failure on clients, e.g. currently all tests run on PRs on the client - so if we don't have a PR for a few days, we might encounter some failure from the server within those days and not catch it. But if the tests run on a cron, we can catch issues on a daily basis.

[VachaShah]

@Yury-Fridlyand has a PR for reducing the time for the integration tests against unreleased OpenSearch, this would significantly reduce the time taken by the workflows on a PR.