Add experimental features flag option for tests

[neetikasinghal]

Description

As of now, there is no way in the tests to turn on the experimental feature flags for the tests, this pr aims to address it.

Issues Resolved

Related prs: #368

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Changes Analysis

Commit SHA: 84f40ca
Comparing To SHA: 83343b2

API Changes

Summary

NO CHANGES

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10204919379/artifacts/1766492962

API Coverage

	Before	After	Δ
Covered (%)	500 (48.97 %)	500 (48.97 %)	0 (0 %)
Uncovered (%)	521 (51.03 %)	521 (51.03 %)	0 (0 %)
Unknown	24	24	0

[dblock]

The code passes generic OPENSEARCH_JAVA_OPTS but claims this is OPENSEARCH_EXPERIMENTAL_FEATURES which it's not, it can be any opts. Get rid of mentions of "experimental features".

An alternate and likely a nicer implementation could be to actually pass a list of experimental features.

In test-spec.yml.

- version: 2.16.0
   features:
     - tiered_remote_index: true

To convert this into OPENSEARCH_JAVA_OPTS you could use join.

[github-actions]

Spec Test Coverage Analysis

Total	Tested
524	120 (22.9 %)

[neetikasinghal]

The code passes generic OPENSEARCH_JAVA_OPTS but claims this is OPENSEARCH_EXPERIMENTAL_FEATURES which it's not, it can be any opts. Get rid of mentions of "experimental features".

An alternate and likely a nicer implementation could be to actually pass a list of experimental features.

In test-spec.yml.

- version: 2.16.0
   features:
     - tiered_remote_index: true

To convert this into OPENSEARCH_JAVA_OPTS you could use join.

I tried the following:

    steps:
      - name: Set OPENSEARCH_JAVA_OPTS
        run: |
          # Initialize options array
          opts=()
          prefix="-Dopensearch.experimental.feature"
          suffix="enabled"
          
          # Extract features directly from the matrix
          features=${{ matrix.entry.features }}
          
          # Split features into key-value pairs
          IFS=',' read -r -a feature_pairs <<< "$features"
          
          # Process each key-value pair
          for pair in "${feature_pairs[@]}"; do
            IFS='=' read -r key value <<< "$pair"
            opts+=("${prefix}.${key}.${suffix}=${value}")
          done
          
          # Set the environment variable
          echo "OPENSEARCH_JAVA_OPTS=$(IFS=','; echo "${opts[*]}")" >> $GITHUB_ENV
          echo $(cat $GITHUB_ENV)

However, its coming as OPENSEARCH_JAVA_OPTS=-Dopensearch.experimental.feature.Array.enabled=

refer: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10189736051/job/28188313025

So, I am going to go with opts for now unless u have another suggestion.

[dblock]

Looks like the updated SHA is bringing in a new feature that's not documented in the spec causing some tests to fail. Either add the new fields (looks easy), or change the SHA back to a previous one for now.

[dblock]

I updated the 2.16 ref to the latest in #459 and tests didn't fail the way they did here, so I looked at the @sha256:50fbfe3b95c41e92a113ada3e80513ba4524dfc8a25dc6aaeff2bbe1e1145d5f SHA and locally all my tests passed. So I added -Dopensearch.experimental.feature.tiered_remote_index.enabled=true and it failed like in this PR. Looks like the cache_utilized and file_cache properties are only enabled with this experimental feature enabled.

[neetikasinghal]

I updated the 2.16 ref to the latest in #459 and tests didn't fail the way they did here, so I looked at the @sha256:50fbfe3b95c41e92a113ada3e80513ba4524dfc8a25dc6aaeff2bbe1e1145d5f SHA and locally all my tests passed. So I added -Dopensearch.experimental.feature.tiered_remote_index.enabled=true and it failed like in this PR. Looks like the cache_utilized and file_cache properties are only enabled with this experimental feature enabled.

Looks like its coming from here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L385

[dblock]

I updated the 2.16 ref to the latest in #459 and tests didn't fail the way they did here, so I looked at the @sha256:50fbfe3b95c41e92a113ada3e80513ba4524dfc8a25dc6aaeff2bbe1e1145d5f SHA and locally all my tests passed. So I added -Dopensearch.experimental.feature.tiered_remote_index.enabled=true and it failed like in this PR. Looks like the cache_utilized and file_cache properties are only enabled with this experimental feature enabled.

Looks like its coming from here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L385

Are you looking into it? Open an issue at least?

[neetikasinghal]

I updated the 2.16 ref to the latest in #459 and tests didn't fail the way they did here, so I looked at the @sha256:50fbfe3b95c41e92a113ada3e80513ba4524dfc8a25dc6aaeff2bbe1e1145d5f SHA and locally all my tests passed. So I added -Dopensearch.experimental.feature.tiered_remote_index.enabled=true and it failed like in this PR. Looks like the cache_utilized and file_cache properties are only enabled with this experimental feature enabled.

Looks like its coming from here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L385

Are you looking into it? Open an issue at least?

yeah checking with @andrross to understand his thought process behind this change - doesnt look super right to me as If the tiered remote index feature flag is on and there is no search node it the cluster it would set the NODE_SEARCH_CACHE_SIZE_SETTING to 80%.

[dblock]

I updated the 2.16 ref to the latest in #459 and tests didn't fail the way they did here, so I looked at the @sha256:50fbfe3b95c41e92a113ada3e80513ba4524dfc8a25dc6aaeff2bbe1e1145d5f SHA and locally all my tests passed. So I added -Dopensearch.experimental.feature.tiered_remote_index.enabled=true and it failed like in this PR. Looks like the cache_utilized and file_cache properties are only enabled with this experimental feature enabled.

Looks like its coming from here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/node/Node.java#L385

Are you looking into it? Open an issue at least?

yeah checking with @andrross to understand his thought process behind this change - doesnt look super right to me as If the tiered remote index feature flag is on and there is not search node it the cluster it would set the NODE_SEARCH_CACHE_SIZE_SETTING to 80%.

For this PR, remove opts: -Dopensearch.experimental.feature.tiered_remote_index.enabled=true from CI and it will go green, so we can then merge it.

In your other PR where you actually need -Dopensearch.experimental.feature.tiered_remote_index.enabled=true we need a fix. If it's in OpenSearch we can wait for it.

[dblock]

The intermittent failure is an OOM in the sql plugin, I changed the threshold in another PR.