Retry remote state download while bootstrap #15950
Conversation
soosinha
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
jainankitk,
kotwanikunal,
linuxpi,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
|
❌ Gradle check result for 3e02411: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Sooraj Sinha <[email protected]>
soosinha
force-pushed
the
retry_bootstrap
branch
from
3e02411
to
8f79550
Compare
|
❕ Gradle check result for 8f79550: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #15950 +/- ##
============================================
- Coverage 71.90% 71.68% -0.23%
+ Complexity 64216 64112 -104
============================================
Files 5272 5277 +5
Lines 300597 300708 +111
Branches 43440 43451 +11
============================================
- Hits 216151 215567 -584
- Misses 66680 67385 +705
+ Partials 17766 17756 -10 ☔ View full report in Codecov by Sentry. |
server/src/main/java/org/opensearch/gateway/GatewayMetaState.java
Outdated
Show resolved
Hide resolved
...rc/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreClusterStateRestoreIT.java
Outdated
Show resolved
Hide resolved
| int delayInMills = 100; | ||
| for (int attempt = 1; attempt <= maxAttempts; attempt++) { | ||
| try { | ||
| return restoreClusterState(remoteStoreRestoreService, clusterState, lastKnownClusterUUID); |
There was a problem hiding this comment.
add a info log for every attempt
There was a problem hiding this comment.
Maybe reuse some retryable entity already present in cluster manager
There was a problem hiding this comment.
@Bukhtawar I do not see a suitable retryable entity already present. Let me know if you have any reference.
Signed-off-by: Sooraj Sinha <[email protected]>
Description
When a remote state enabled cluster manager node boots-up, it tries to download the remote state. But if there some issue while downloading like file not present then the download fails with an exception. The OpenSearch process stays active but unresponsive. This PR addresses this by:
Errorso that process exists and a new process is spawned.Related Issues
NA
Check List
API changes companion pull request created, if applicable.Public documentation issue/PR created, if applicable.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.