Improve "SENTINEL FAILOVER" by using the "FAILOVER" command #1292
+484
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Simon Baatz <[email protected]>
Up to now the sentinel test initialization verified that all sentinels detect each other. However, detection does not imply connection, which led to intermittent failures in the coordinated failover tests (no leader elected since disconnected sentinels do not take part in a vote). Fix this by waiting until no sentinel reports being "disconnected". Signed-off-by: Simon Baatz <[email protected]>
Valkey supports the "FAILOVER" command to switch primary and replica roles in a coordinated fashion. Add a "COORDINATED" option to "SENTINEL FAILOVER". When given, use "FAILOVER" in the Sentinel forced failover procedure. Since we force the failover, i.e. we don't elect a leader, the failover must succeed quickly before other Sentinels will regard the primary as down. Signed-off-by: Simon Baatz <[email protected]>
The FAILOVER command may ["encounter some scenarios it can not automatically remediate from and may get stuck."](https://valkey.io/commands/failover/#failover-abort). The supervision of the Sentinels is used to handle these situations: 1. "REPLICAOF" is not accepted during a failover. Thus, send a "FAILOVER ABORT" before sending a "REPLICAOF" to a node in `sentinelSendReplicaOf()`. (If there is no ongoing failover, the resulting error will just be ignored) 2. Sentinel monitors nodes for deviations from the expected state (wrong role or replication). Add a check for replicas that are in a failover state for too long and reconfigure them. Signed-off-by: Simon Baatz <[email protected]>
Doing a proper leader election allows to increase the time available for the actual FAILOVER command, as other Sentinels will not attempt to initiate another failover during the failover timeout. Interestingly, Sentinel does not take the reported primary up/down status into account when counting replies to leader election. Thus, Sentinel will simply proceed with the failover once we reach the quorum even if we don't are in O_DOWN state. Signed-off-by: Simon Baatz <[email protected]>
This was referenced Nov 12, 2024
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #1292 +/- ##
============================================
- Coverage 70.69% 70.62% -0.07%
============================================
Files 114 114
Lines 63161 63247 +86
============================================
+ Hits 44650 44667 +17
- Misses 18511 18580 +69
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements #1291 and consists of the following commits:
Two commits to fix problems with the sentinel tests:
Commit: Wait for all Sentinels to be connected before starting tests
Up to now the sentinel test initialization verified that all sentinels detect each other.
However, detection does not imply connection, which led to intermittent failures in the
coordinated failover tests (no leader elected since disconnected sentinels do not take part in a
vote).
Fix this by waiting until no sentinel reports being "disconnected".
Commit: sentinel-tests: Clean up config after config set tests
Three commits with the actual implementation:
Commit: Add option for coordinated failover to Sentinel (coordinated failover without leader election)
Commit: Allow Sentinel to recover from a stuck FAILOVER
Commit: SENTINEL FAILOVER COORDINATED actually does a leader election (adds leader election before failover starts)