Investigate why the Chainhook chain segment can become outdated

[csgui]

An issue has happened, in production, where the Chainhook’s chain segment became outdated.

And since the code base is not prepared to handle that, only restoring the Chainhook's chain state solves this issue.

https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L255
https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L414

The following log messages where triggered in the case of that situation.

Will have to fork: OutdatedSegment
Unable to process Stacks Block #158000 (0x089f...445b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157999 (0x2ee0...087b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157998 (0x4353...70af) - inboxed for later
unable to infer chain progress

The work here is to find the root cause for why this can happen.

[smcclellan]

I see two issues here:

1. A spike to investigate why the chain state became outdated, ideally reproducing the issue
2. Make the system more resilient so it can recover from an outdated chain state

Let's have this ticket be the first one, and I'll create a separate ticket for the second one.