[1.0-beta3] avoid unnecessary state history log "unrotations" that can remove retained logs on replay/resync

[spoonincode]

In the early days, ship would truncate its log any time the applied block number was less than the log's head block. But this could lead to substantial chunks of the log being removed in the case of a replay or resync (such as when restarting from a snapshot).¹

So instead, sometime in leap, the log behavior was modified such that when the applied block number decreases relative to the head of the log, the log would check if the blockid in the log matched that as the new applied block. If it matched, it'd do nothing, since that is indicative of a replay or resync. If it did not match, the log would be truncated as before, since that is indicative of a new fork being applied.

The log rewrite in spring 1.0 maintains this strategy. However, the blockid duplication check is performed in the log, not the log_catalog. This means for a partitioned log where a block is being applied in a retained log the log_catalog would always unrotate any higher numbered logs before figuring out it doesn't need to "truncate" (append now in spring1) the log.

So, add a duplication check in log_catalog too to guard against this. And add a handful of tests that are in the same ballpark as replay/resync (writing identical blocks to the log again).

(it is a little uneasy to see some code duplication in log and log_catalog. The original idea was that log and log_catalog would share the same public interface such that if nodeos was configured without log splitting it could simply use log directly -- maybe via some template even since the interface would completely match up. This commonality of interface still remains: pack_and_write_entry(), get_entry(), get_block_id(), block_range(), and empty() all have an identical interface between the two. But there was nothing ever attempted to strictly use log if that's all that is required)

Footnotes

1. There also could be disastrous consequences if accidentally restarting from genesis that would erase the entire log. That is guarded against differently than what is further discussed here. ↩

[heifner]

I don't think we need to add this feature now, but I have had to provide a patch in the past to bypass this behavior so that a corrupted SHiP log could be repaired by replaying from a snapshot before the corrupted log.

[spoonincode]

yep that reminds me of AntelopeIO/leap#2393

Probably we should add something to spring-util to allow simple mutations of the log similar to the block log.

[ericpassmore]

Note:start
group: STABILITY
category: INTERNALS
summary: Improved block id duplication logic to retain the correct state history with replay/resync.
Note:end