[Backport 2.11] Fix shard failure on flush during upload failures for remote indexes #10587
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
….JSON (opensearch-project#9156) (opensearch-project#9210) This commit rote refactors nearly all instances and usages of the XContentType.JSON singleton instance defined in the :libs:opensearch-x-content library to the MediaTypeRegistry.JSON singleton instance defined in :libs:opensearch-core. This decouples the server and test implementations from the x-content library such that foundation classes can be further refactored from the :server module to the :opensearch-core library without requiring a strict dependency on the x-content library. This keeps opensearch-core library loosely coupled with the remaining :libs:opensearch-* libraries such that serverless and cloud-native implementations can selective import only the libraries needed for implementation. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit 7278f43) --------- Signed-off-by: Nicholas Walter Knize <[email protected]>
Signed-off-by: Kunal Kotwani <[email protected]>
…search-project#9170) Signed-off-by: Sarthak Aggarwal <[email protected]> (cherry picked from commit 3713bcb)
…/repository-azure (opensearch-project#9231) (opensearch-project#9260) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 1162865) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ect#9271) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (opensearch-project#9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 48ec206) * Force upgrade the vulnerable dependencies of hadoop-minicluster (opensearch-project#9252) * Force upgrade the vulnerable dependencies of hadoop-minicluster Signed-off-by: Thomas Farr <[email protected]> * Add changelog entry Signed-off-by: Thomas Farr <[email protected]> --------- Signed-off-by: Thomas Farr <[email protected]> (cherry picked from commit a1fc31c) * Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 in /plugins/repository-gcs (opensearch-project#9230) * Bump com.google.code.gson:gson in /plugins/repository-gcs Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.9.0 to 2.10.1. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10.1) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 5c283e0) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (opensearch-project#9228) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.2.0 to 1.8.0. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v1.2.0...v1.8.0) --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 4b21849) --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas Farr <[email protected]>
…fix PIT/Scroll with remote store. (opensearch-project#9272) * [Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <[email protected]> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <[email protected]> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Add more shard level tests. Signed-off-by: Marc Handalian <[email protected]> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <[email protected]> * Self review. Signed-off-by: Marc Handalian <[email protected]> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit c301544) * Fix test SegmentReplicationIndexShardTests.testPrimaryRestart. This test is specific to remote store and should not be run for node-node replication. Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit a33f67e) Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
…project#9274) (cherry picked from commit 3ddff4c) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…-project#9244) (opensearch-project#9273) (cherry picked from commit ab08d73) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…est (opensearch-project#9079) (opensearch-project#9278) * Unmute testIndexingWithSegRep rolling upgrade test * Assert on row count before processing the result * Filter rows with 0 doc count * Retry assertHitCount * Handle exception * Add comment --------- Signed-off-by: Suraj Singh <[email protected]>
…) (opensearch-project#9282) (cherry picked from commit db768d1) Signed-off-by: Jay Deng <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…andle restore … (opensearch-project#9261) --------- Signed-off-by: bansvaru <[email protected]>
…arch-project#8758) (opensearch-project#9268) Signed-off-by: Ashish Singh <[email protected]>
…search-project#9122) (opensearch-project#9275) * Make SearchTemplateRequest implement IndicesRequest.Replaceable * Add to CHANGELOG --------- (cherry picked from commit c73f727) Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…oject#9292) * Adds attributes to startSpan * Update Changelog * Adds attributes to startSpan * Refactor code * Add java doc * Refactor code * Refactor code * Refactor code * Removes dependency --------- Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]>
opensearch-project#9293) (opensearch-project#9295) --------- (cherry picked from commit 5fdebee) Signed-off-by: bansvaru <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…annotations, their meaning and relations between them (opensearch-project#9223) (opensearch-project#9296) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (opensearch-project#9223) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> (cherry picked from commit 0b2c07a) * [BWC and API enforcement] Extend the element types the initial set of annotations could be applied to Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]>
…rojects) (opensearch-project#9309) (opensearch-project#9319) Signed-off-by: Owais Kazi <[email protected]> (cherry picked from commit 301c766) Co-authored-by: Owais Kazi <[email protected]>
…hmarks/ subprojects) (opensearch-project#9307) (opensearch-project#9313) (cherry picked from commit 8cecd5a) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…dSrc/ subprojects) (opensearch-project#9314) (opensearch-project#9329) (cherry picked from commit 4798502) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…rch-project#9324) (opensearch-project#9334) (cherry picked from commit 9dca96d) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…ensearch-project#9323) (opensearch-project#9333) (cherry picked from commit 64be0ab) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…-project#9082) (opensearch-project#9352) This is the first round refactor of the foundation Task classes from the server module to core and common library. This is split into separate commits due to control the surface area impacted by the change. This commit refactors the following classes: * o.o.action.ActionFuture -> :libs:opensearch-common * o.o.action.ActionListener -> :libs:opensearch-core * o.o.action.ActionResponse -> :libs:opensearch-core * o.o.action.NotifyOnceListener -> :libs:opensearch-core * o.o.common.compress.Compressor -> :libs:opensearch-core * o.o.node.ReportingService -> o.o.core.service in :libs:opensearch-core * o.o.tasks.TaskCancelledExceptoin -> :libs:opensearch-core * o.o.tasks.TaskId -> :libs:opensearch-core * o.o.common.xcontent.ObjectParserHelper -> :libs:opensearch-core The following task resource tracking classes are refactored into a new resourcetracker namespace under the o.o.core.tasks packed in :libs:opensearch-core * o.o.tasks.ResourceStats * o.o.tasks.ResourceStatsType * o.o.tasks.ResourceUsageMetric * o.o.tasks.TaskResourceStats * o.o.tasks.TaskResourceUsage * o.o.tasks.TaskThreadUsage * o.o.tasks.ThreadResourceInfo These refactors are part of the effort to split foundation classes into core libraries in support of modularity for serverless and cloud native implementations. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit de2b6b7)
…earch-project#9347) Guava is a transitive dependency of spotless via google-java-format, the version that was being pulled in has CVE-2023-2976. While this would not affect end users it causes excess alerts. (cherry picked from commit 1342578) Signed-off-by: Thomas Farr <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…opensearch-project#9354) (opensearch-project#9358) This fails frequently. Muting for now until it is fixed to unblock unrelated PRs. (cherry picked from commit 4114009) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…oject#9302) (opensearch-project#9311) * Bump aws-actions/configure-aws-credentials from 1 to 2 Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 1 to 2. - [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases) - [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md) - [Commits](aws-actions/configure-aws-credentials@v1...v2) --- updated-dependencies: - dependency-name: aws-actions/configure-aws-credentials dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit a579cb9) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…nt/ subprojects) (opensearch-project#9345) (opensearch-project#9359) Signed-off-by: Andriy Redko <[email protected]>
…les/ subprojects) (opensearch-project#9346) (opensearch-project#9361) Signed-off-by: Andriy Redko <[email protected]>
…roject#9280) (opensearch-project#9357) (cherry picked from commit 23fd9a5) Signed-off-by: Peter Nied <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…rojects) (opensearch-project#9351) (opensearch-project#9370) Signed-off-by: Owais Kazi <[email protected]> (cherry picked from commit a1faafe) Co-authored-by: Owais Kazi <[email protected]>
…pensearch-project#10513) Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
ashking94
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal,
mch2,
nknize,
owaiskazi19,
peternied,
Rishikesh1159,
ryanbogan,
saratvemulapalli,
shwetathareja,
dreamer-89,
VachaShah,
dbwiddis,
sachinpkale,
sohami and
msfroh
as code owners
|
Mistakenly raised to main was wanting to open against 2.11. |
Compatibility status:Checks if related components are compatible with change c306908 Incompatible componentsSkipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git] |
Gradle Check (Jenkins) Run Completed with:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 90c4297 from #10513.