Prevent unnecessary lib<->head catchup transition changes

[linh2931]

Resolved #1546

Replace #1617 to base on main instead.

Instant finality or private chains make it likely peers think their lib and head are not in sync but in reality they are only within small difference. To avoid unnecessary catchups, a minimal distance between lib and head is introduced. min_blocks_distance must be reached before catchup starts.

min_blocks_distance is defined as the number of blocks produced during the half keep alive interval (how long a node waits before contacting its peer).

[heifner]

Hold up. I want to take a look at the test failure.

[heifner]

The test is failing because the node that is producing believes the other node is syncing from it so it does not send it the block it produced.

I think we need to revert this change (chain_info.head_num + min_blocks_distance < msg.last_irreversible_block_num) while keeping the rest. That likely means that more handshake messages will be sent. That handshake should be seen from the peer as having the same head and drop out of syncing.

One thing we can look at doing in the future is using the hotstuff messages as a way to know if we are insync. They include the current block_id, so if we have that block then we know we are insync. We can therefore know we are insync without the extra handshake message.

[linh2931]

@heifner Thank you so much for your help!