ZKR_DB_KEY_NOT_FOUND while proving batch 75948 on Bali

[ToniRamirezM]

It looks like an issue with the witness generation. It was detected on CDK-Erigon v2.60.0-beta9.

https://app.datadoghq.com/logs?query=host%3Agke-zkevm-bali-pool-prover-8de10c2f-tgnj.europe-west1-d.c.prj-polygonlabs-zkevm-test.internal%20service%3Azkevm-bali%20container_id%3Abb3ef65d6725c7bde954e60429e7e1384fa290cddc5ec20e034910f770c2a0aa%20filename%3A1.log&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&context_event=AZMmpCZWAAAq-BxZD_qnuQAK&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Cdesc&to_event=AwAAAZMmpCNv1GOUtgAAABhBWk1tcENaV0FBQXEtQnhaRF9xbnVRQVIAAAAkMDE5MzI2YTctNzU1Zi00NmMzLTkxNTQtZDU4OWY0YjI2ODE2AAAUaw&viz=&from_ts=1731519811468&to_ts=1731520111472&live=false

[hexoscott]

Some notes from the investigation so far:
The problem is with batch 75948 - looking at the batchL2Data for this batch there is no L1 info tree index change. This is important because the missing slot is related to writing the L1 Block Hash to the GER manager contract. This is only written to if we use a new index at the sequencer.

The prover logs show this as the missing information from the witness:

A=0:0:0:a40d5f56:745a118d:906a34e:69aec8c0:db1cb8fa B=0:0:0:0:0:0:0:3 C=17f78b8d:30f15ffe:e2432660:db3ab8af:ebdac70d:db400edd:b329e010:4952e40e D=0:0:0:0:0:0:0:20 E=0:0:0:0:0:0:0:58

I ran a binary search for when this slot was changed on Bali and it was updated in block 6958864. This block belongs to batch 74981, so 966 batches prior to the one the prover is having trouble with which seems strange that it would need this slot at all.

My hunch at the moment is that this issue is somehow related to the info tree index changes. The problem on the prover stems from the changeL2Block routine which seems to confirm this theory.