Restarting during initial genesis sync resets block headers

[quickchase]

System information

Erigon version: erigon version 2.43.0-dev-38441b67

OS & Version: Linux

The node is syncing from scratch (genesis) on Chapel, it starts, it starts syncing headers, it's then restarted, and most of its progress writing block headers is apparently not saved between restarts.

Here's the logs, partially truncated. This was straight systemctl restart ergion:

...
[2/15 Headers] Wrote block headers       number=2537139 blk/second=190.000 alloc=281.8MB sys=411.4MB
[2/15 Headers] Wrote block headers       number=2540749 blk/second=180.500 alloc=352.0MB sys=411.5MB
[2/15 Headers] Wrote block headers       number=2544359 blk/second=180.500 alloc=251.9MB sys=411.5MB
[2/15 Headers] Wrote block headers       number=2547969 blk/second=180.500 alloc=332.5MB sys=411.5MB
[2/15 Headers] Wrote block headers       number=2551579 blk/second=180.500 alloc=223.7MB sys=416.6MB
[2/15 Headers] Wrote block headers       number=2555189 blk/second=180.500 alloc=285.1MB sys=416.6MB
...
Got interrupt, shutting down...
Exiting Engine...
RPC server shutting down
Exiting...
...
Build info                               git_branch=HEAD git_tag=v1.1.0 git_commit=38441b6774033ac2ddf729bd4322a706fe90d33a
Starting Erigon on Chapel testnet...
...
Initialised chain configuration          config="{ChainID: 97 Ramanujan: 1010000, Niels: 1014369, MirrorSync: 5582500, Bruno: 13837000, Euler: 19203503, Gibbs: 22800220, Nano: 23482428, Moran: 23603940, Gibbs: 22800220, Planck: 28196022, Luban: 29295050, Plato: 29861024, Engine: parlia}" genesis=0x6d3c66c5357ec91d5c43af47e234a939b22557cbb552dc45bebbceeed90fbe34
...
Effective                                prune_flags= snapshot_flags= history.v3=false
Initialising Ethereum protocol           network=97
[downloader] recover headers from db     left=65536
...
[2/15 Headers] Waiting for headers...    from=1278199
[2/15 Headers] Wrote block headers       number=1281809 blk/second=180.500 alloc=65.0MB sys=112.8M
...

Somehow during the restart of the service, it lost ~1,276,990 block headers...nearly half of its progress.

[nmiceli-simtlix]

I am not sure if this is related. I have 2 nodes syncing from scratch on chapel as well. Everything seemed to be working fine until both tried to execute block 1278199

I am running version 1.1.0

[INFO] [06-12|21:15:18.069] [7/15 Execution] Executed blocks         number=1278103 blk/s=128.1 tx/s=297.9 Mgas/s=189.1 gasState=0.27 batch=122.4MB alloc=3.3GB sys=7.6GB
[WARN] [06-12|21:15:19.190] [7/15 Execution] Execution failed        block=1278200 hash=0xd2ee744a60ccf9b59cd928f9d237dba89e1e6f9140f9af1ab873dc3ea4148903 err="mismatching validator list on epoch block"
[INFO] [06-12|21:15:19.190] UnwindTo                                 block=1278199 bad_block_hash=0xd2ee744a60ccf9b59cd928f9d237dba89e1e6f9140f9af1ab873dc3ea4148903
[INFO] [06-12|21:15:30.982] [7/15 Execution] Completed on            block=1278199
[INFO] [06-12|21:15:30.982] [7/15 Execution] DONE                    in=35m33.116998382s

[setunapo]

there is some update on how to get the validator list, it is on the latest develop branch, we will do another release, you may try later.

[nmiceli-simtlix]

Upgrading to v1.1.1 fixed the issue for block 1278199.
Now I am getting the same error on block 29517599

[WARN] [06-17|22:32:41.260] [7/15 Execution] Execution failed        block=29517600 hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 err="mismatching validator list on epoch block"
[INFO] [06-17|22:32:41.260] UnwindTo                                 block=29517599 bad_block_hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2
[INFO] [06-17|22:33:28.387] [7/15 Execution] Completed on            block=29517599

[calmbeing]

@quickchase @nmiceli-simtlix should be fixed by this pr #135