Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.0] Pause production when max-reversible-blocks reached #652

Merged
merged 4 commits into from
Aug 27, 2024

Conversation

heifner
Copy link
Member

@heifner heifner commented Aug 27, 2024

Change the behavior of max-reversible-blocks. Instead of shutting down a node when max-reversible-blocks is reached; pause production.

  • A node will no longer shutdown when it reaches max-reversible-blocks.
  • A block production node will not produce additional blocks when max-reversible-blocks is reached.

A block production node will report an error: Not producing block because max-reversible-blocks ${m} reached, head ${h}, lib ${l}. and will instead speculatively execute transactions until LIB advances. Note LIB can advance by finalizers being restored and other block producers producing blocks. Once the number of reversible blocks is reduced on the network, the block producer will un-pause and start producing blocks again.

If all block producers have the same default value for max-reversible-blocks (3600) then after approximately 30 minutes of LIB not advancing, all block producers will pause production. Manual restart of at least one block producer will be required at that point, after finality issues have been resolved, with a higher max-reversible-blocks. Once a block producer with network finalizers are able to advance finality the number of reversible blocks should be reduced back down to ~2 and all other block producers will automatically un-pause and begin producing again.

Resolves #641
Resolves #634

@heifner heifner added the OCI Work exclusive to OCI team label Aug 27, 2024
@heifner heifner linked an issue Aug 27, 2024 that may be closed by this pull request
Base automatically changed from GH-635-sync-ahead to release/1.0 August 27, 2024 14:43
@@ -1976,6 +1982,11 @@ producer_plugin_impl::start_block_result producer_plugin_impl::start_block() {
("ov", _last_other_vote_received.load(std::memory_order_relaxed))("bt", _accepted_block_time));
_pending_block_mode = pending_block_mode::speculating;
not_producing_when_time = true;
} else if (head_block_num - head.irreversible_blocknum() >= _max_reversible_blocks) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should >= be > as the option description says beyond?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check _max_reversible_blocks not 0. Otherwise always pause.

@@ -1976,6 +1982,11 @@ producer_plugin_impl::start_block_result producer_plugin_impl::start_block() {
("ov", _last_other_vote_received.load(std::memory_order_relaxed))("bt", _accepted_block_time));
_pending_block_mode = pending_block_mode::speculating;
not_producing_when_time = true;
} else if (head_block_num - head.irreversible_blocknum() >= _max_reversible_blocks) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check _max_reversible_blocks not 0. Otherwise always pause.

@heifner heifner merged commit 511eabe into release/1.0 Aug 27, 2024
36 checks passed
@heifner heifner deleted the GH-641-max-reversible-blocks-block-production branch August 27, 2024 16:55
@heifner heifner linked an issue Aug 27, 2024 that may be closed by this pull request
@ericpassmore
Copy link
Contributor

Note:start
group: STABILITY
category: INTERNALS
summary: Change behavior to pause production when max-reversible-blocks are reached.
Note:end

@ericpassmore ericpassmore added the bug The product is not working as was intended. label Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug The product is not working as was intended. OCI Work exclusive to OCI team
Projects
None yet
4 participants