Add allowlist setting for ingest-common processors #14479

andrross · 2024-06-20T20:46:42Z

Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows:

If the allowlist setting is not defined, all installed processors are enabled. This is the status quo.
If the allowlist setting is defined as the empty set, then all processors are disabled.
If the allowlist setting contains the names of valid processors, only those processors are enabled.
If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed.
If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts.

Related to #14439

Check List

Functionality includes testing.
~~API changes companion pull request created, if applicable.~~
Public documentation issue/PR created, if applicable: Document new ingest and search pipeline allowlist settings documentation-website#7414

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-06-20T21:39:02Z

❕ Gradle check result for 1722742: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-06-20T21:45:45Z

✅ Gradle check result for a4d3675: SUCCESS

server/src/main/java/org/opensearch/ingest/IngestService.java

server/src/test/java/org/opensearch/ingest/IngestServiceTests.java

github-actions · 2024-06-20T23:58:24Z

❌ Gradle check result for f15a5c0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

codecov · 2024-06-25T21:53:10Z

Codecov Report

Attention: Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.71%. Comparing base (8839904) to head (72cb90c).
Report is 1 commits behind head on main.

Files	Patch %	Lines
...search/ingest/common/IngestCommonModulePlugin.java	94.73%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14479      +/-   ##
============================================
+ Coverage     71.65%   71.71%   +0.05%     
+ Complexity    62174    62165       -9     
============================================
  Files          5125     5125              
  Lines        292412   292428      +16     
  Branches      42254    42255       +1     
============================================
+ Hits         209541   209709     +168     
+ Misses        65625    65375     -250     
- Partials      17246    17344      +98

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonModulePlugin.java

gaobinlong · 2024-06-26T05:27:41Z

Should we prevent users from not allowing the processors which are used by existing ingest pipelines? Indexing failures will happen if users change the setting but some in-use processors are not in the allowlist.

reta · 2024-06-26T12:44:07Z

Should we prevent users from not allowing the processors which are used by existing ingest pipelines? Indexing failures will happen if users change the setting but some in-use processors are not in the allowlist.

I think this is one of the goals - if the processor could be disabled at any time (for whatever reasons). The same would happen if plugin with custom processors is uninstalled.

Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]>

github-actions · 2024-06-26T16:28:02Z

❌ Gradle check result for 72cb90c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-26T18:29:12Z

❕ Gradle check result for 72cb90c: UNSTABLE

TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testDeleteBlobs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

owaiskazi19

Looks good overall with minor comments

modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonModulePlugin.java

Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to #14439 Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit a99b494) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…4561) * Add allowlist setting for ingest-common processors (#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to #14439 Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit a99b494) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix test issues due to class renaming on main Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <[email protected]>

…t#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

@internalapi

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <[email protected]> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <[email protected]> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <[email protected]> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <[email protected]> * Added changelog Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments Signed-off-by: Sagar Upadhyaya <[email protected]> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <[email protected]> * minor UT refactor Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <[email protected]> * Update TESTING.md Co-authored-by: Marc Handalian <[email protected]> Signed-off-by: bowenlan-amzn <[email protected]> * Address comment Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Co-authored-by: Marc Handalian <[email protected]> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * [AUTO] [main] Add bwc version 2.15.1. (opensearch-project#14549) * Add bwc version 2.15.1 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix auto-generated version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550) * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently Signed-off-by: Sagar Upadhyaya <[email protected]> * Addressing comment Signed-off-by: Sagar Upadhyaya <[email protected]> --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Fix file cache initialization (opensearch-project#14004) * fix file cache initialization Signed-off-by: panguixin <[email protected]> * changelog Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> * Add Ashish Singh as maintainer (opensearch-project#14567) Signed-off-by: Bukhtawar Khan <[email protected]> * Allow @internalapi annotation on classes not meant to be constructed outside of the OpenSearch core (opensearch-project#14575) Signed-off-by: Andriy Redko <[email protected]> * Bump com.azure:azure-storage-common from 12.21.2 to 12.25.1 in /plugins/repository-azure (opensearch-project#14517) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.21.2 to 12.25.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-common_12.21.2...azure-storage-blob_12.25.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Add allowlist setting for search-pipeline-common processors (opensearch-project#14562) Add a new static setting that lets an operator choose specific search pipeline processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. A distinct setting exists for each of request, response, and search phase results processors. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> * Bump Apache Lucene to 9.11.1 (opensearch-project#14576) (opensearch-project#14581) (cherry picked from commit 0095fd1) Signed-off-by: Andriy Redko <[email protected]> * Add unittests for RemoteClusterStateAttributesManager (opensearch-project#14427) * Add unittests for RemoteClusterStateAttributesManager Signed-off-by: Shivansh Arora <[email protected]> * Add Ashish Singh to codeowners (opensearch-project#14592) Signed-off-by: Ashish Singh <[email protected]> * Add batching processor base type AbstractBatchingProcessor (opensearch-project#14554) Signed-off-by: Liyun Xiu <[email protected]> * Add @internalapi annotation to japicmp exclusions (opensearch-project#14597) Signed-off-by: Andriy Redko <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is mis… (opensearch-project#14552) * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Parsing a GetResult returns NPE if found field is missing. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix wildcart import. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:Fix spotless issues. Signed-off-by: Vatsal <[email protected]> Signed-off-by: vatsal <[email protected]> * Fix issue 14519:update changelog Signed-off-by: vatsal <[email protected]> --------- Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> * Star tree mapping changes (opensearch-project#14605) * Star tree mapping changes with feature flag --------- Signed-off-by: Bharathwaj G <[email protected]> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <[email protected]> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <[email protected]> * rerun security check Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <[email protected]> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <[email protected]> * Add changelog Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove unused logger Signed-off-by: Siddhant Deshmukh <[email protected]> * Spotless apply Signed-off-by: Siddhant Deshmukh <[email protected]> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <[email protected]> --------- Signed-off-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <[email protected]> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <[email protected]> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <[email protected]> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <[email protected]> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed nits Signed-off-by: Sarthak Aggarwal <[email protected]> * addressed major nits Signed-off-by: Sarthak Aggarwal <[email protected]> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <[email protected]> * handling for missing doc values Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * rebasing with main Signed-off-by: Sarthak Aggarwal <[email protected]> * support for empty sequential doc values iterator Signed-off-by: Sarthak Aggarwal <[email protected]> * nits Signed-off-by: Sarthak Aggarwal <[email protected]> * min and max star tree aggregators Signed-off-by: Sarthak Aggarwal <[email protected]> * star tree file formats * Star tree codec changes Signed-off-by: Bharathwaj G <[email protected]> * Adding tests Signed-off-by: Bharathwaj G <[email protected]> * Addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing review comments Signed-off-by: Bharathwaj G <[email protected]> * Star tree merge changes Signed-off-by: Bharathwaj G <[email protected]> * fix annotations * star-tree file formats reader and javadoc fixes * read for composite index values Signed-off-by: Sarthak Aggarwal <[email protected]> * doc values file format Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: Prudhvi Godithi <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: kkewwei [email protected] Signed-off-by: kkewwei <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: panguixin <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Liyun Xiu <[email protected]> Signed-off-by: vatsal <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]> Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Bharathwaj G <[email protected]> Co-authored-by: bowenlan-amzn <[email protected]> Co-authored-by: Prudhvi Godithi <[email protected]> Co-authored-by: Sagar <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]> Co-authored-by: kkewwei <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Co-authored-by: panguixin <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Shivansh Arora <[email protected]> Co-authored-by: Ashish Singh <[email protected]> Co-authored-by: Liyun Xiu <[email protected]> Co-authored-by: Vatsal <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Siddhant Deshmukh <[email protected]> Co-authored-by: Robin Friedmann <[email protected]> Co-authored-by: rishavz_sagar <[email protected]> Co-authored-by: Sooraj Sinha <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]>

…t#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]>

…ensearch-project#14561) * Add allowlist setting for ingest-common processors (opensearch-project#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit a99b494) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix test issues due to class renaming on main Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: kkewwei <[email protected]>

…t#14479) Add a new static setting that lets an operator choose specific ingest processors to enable by name. The behavior is as follows: - If the allowlist setting is not defined, all installed processors are enabled. This is the status quo. - If the allowlist setting is defined as the empty set, then all processors are disabled. - If the allowlist setting contains the names of valid processors, only those processors are enabled. - If the allowlist setting contains a name of a processor that does not exist, then the server will fail to start with an IllegalStateException listing which processors were defined in the allowlist but are not installed. - If the allowlist setting is changed between server restarts then any ingest pipeline using a now-disabled processor will fail. This is the same experience if a pipeline used a processor defined by a plugin but then that plugin were to be uninstalled across restarts. Related to opensearch-project#14439 Signed-off-by: Andrew Ross <[email protected]>

andrross requested review from anasalkouz, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners June 20, 2024 20:46

andrross force-pushed the ingest-allowlist branch from 1722742 to a4d3675 Compare June 20, 2024 20:54

andrross force-pushed the ingest-allowlist branch from a4d3675 to f15a5c0 Compare June 20, 2024 23:03

andrross commented Jun 20, 2024

View reviewed changes

server/src/main/java/org/opensearch/ingest/IngestService.java Outdated Show resolved Hide resolved

andrross commented Jun 20, 2024

View reviewed changes

server/src/test/java/org/opensearch/ingest/IngestServiceTests.java Outdated Show resolved Hide resolved

This was referenced Jun 21, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

[AUTOCUT] Gradle Check Flaky Test Report for ClusterRerouteIT #14298

Open

reta mentioned this pull request Jun 21, 2024

[FEATURE] Add support for experimental codecs opensearch-project/custom-codecs#148

Open

prudhvigodithi mentioned this pull request Jun 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

opensearch-ci-bot mentioned this pull request Jun 26, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

gaobinlong reviewed Jun 26, 2024

View reviewed changes

modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonModulePlugin.java Outdated Show resolved Hide resolved

reta approved these changes Jun 26, 2024

View reviewed changes

andrross force-pushed the ingest-allowlist branch from 8b5c0b5 to 72cb90c Compare June 26, 2024 15:07

This was referenced Jun 26, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteCloneIndexIT #14292

Open

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

saratvemulapalli approved these changes Jun 26, 2024

View reviewed changes

owaiskazi19 reviewed Jun 26, 2024

View reviewed changes

modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonModulePlugin.java Show resolved Hide resolved

modules/ingest-common/src/main/java/org/opensearch/ingest/common/IngestCommonModulePlugin.java Show resolved Hide resolved

owaiskazi19 approved these changes Jun 26, 2024

View reviewed changes

andrross merged commit a99b494 into opensearch-project:main Jun 26, 2024
37 of 38 checks passed

andrross deleted the ingest-allowlist branch June 26, 2024 19:48

opensearch-trigger-bot bot mentioned this pull request Jun 26, 2024

[Backport 2.x] Add allowlist setting for ingest-common processors #14561

Merged

finnegancarroll mentioned this pull request Jul 15, 2024

Make remote snapshot (local)block size configurable #14753

Closed

3 tasks

saratvemulapalli mentioned this pull request Aug 20, 2024

Adding allowlist setting for ingest-useragent and ingest-geoip processors #15325

Merged

2 tasks

opensearch-ci-bot mentioned this pull request Oct 1, 2024

[AUTOCUT] Gradle Check Flaky Test Report for ReplicaToPrimaryPromotionIT #16160

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add allowlist setting for ingest-common processors #14479

Add allowlist setting for ingest-common processors #14479

andrross commented Jun 20, 2024 •

edited

Loading

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

codecov bot commented Jun 25, 2024 •

edited

Loading

gaobinlong commented Jun 26, 2024

reta commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

owaiskazi19 left a comment

Add allowlist setting for ingest-common processors #14479

Add allowlist setting for ingest-common processors #14479

Conversation

andrross commented Jun 20, 2024 • edited Loading

Check List

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

codecov bot commented Jun 25, 2024 • edited Loading

Codecov Report

gaobinlong commented Jun 26, 2024

reta commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

owaiskazi19 left a comment

Choose a reason for hiding this comment

andrross commented Jun 20, 2024 •

edited

Loading

codecov bot commented Jun 25, 2024 •

edited

Loading