Optimize remote store GC flow with pinned timestamps

[sachinpkale]

Description

• Added a bunch of optimisations in remote store garbage collection flow to work with pinned timestamps
  • If index is deleted and a snapshot containing the index is getting deleted:
    • If this is the last snapshot containing the index, delete remote store data (segments + translog)
    • else pass pinning entity for the snapshot that is getting deleted. This is required to delete the data specific to snapshot even if pinned timestamp scheduler runs later.
  • For remote translog, avoid calling list metadata files if minRemoteGenReferenced - indexSettings().getRemoteTranslogExtraKeep() <= maxDeletedGenerationOnRemote
  • For remote translog, make sure to keep metadata and generation deletion in sync. With this, if metadata file is present, we can ensure that generations corresponding to it are not deleted.
  • Added integ tests around these scenarios.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 609bd01: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3186ffa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f07d64d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c344045: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for eb12d3c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a4dab7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 5ed7af5: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 48.14815% with 42 lines in your changes missing coverage. Please review.

Project coverage is 72.02%. Comparing base (3937ccb) to head (fae9f37).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
...index/translog/RemoteFsTimestampAwareTranslog.java	44.00%	24 Missing and 4 partials ⚠️
...ch/repositories/blobstore/BlobStoreRepository.java	50.00%	6 Missing ⚠️
...earch/index/store/RemoteSegmentStoreDirectory.java	50.00%	1 Missing and 2 partials ⚠️
...ex/translog/transfer/TranslogTransferMetadata.java	40.00%	3 Missing ⚠️
...ava/org/opensearch/snapshots/SnapshotsService.java	50.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15943      +/-   ##
============================================
+ Coverage     71.90%   72.02%   +0.12%     
- Complexity    64392    64434      +42     
============================================
  Files          5278     5280       +2     
  Lines        300877   300908      +31     
  Branches      43478    43483       +5     
============================================
+ Hits         216351   216744     +393     
+ Misses        66747    66396     -351     
+ Partials      17779    17768      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for 024a004: SUCCESS

[github-actions]

❌ Gradle check result for 5aaa36e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4b128d1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 22187b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 36eba58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 36eba58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for fae9f37: SUCCESS