bug: Value missing for key AppHash Error

[aditya-manit]

What is happening?

Getting wrong Block.Header.AppHash after this

5:56PM ERR panic recovered in runTx err="recovered: Value missing for key [0 0 0 0 1 110 14 115 0 0 0 1] corresponding to nodeKey 7300000000016e0e7300000001\nstack:\ngoroutine 301 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x31abca0, 0xc013a8a220})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:74 +0x25\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x31abca0?, 0xc013a8a220?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:42 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x31abca0, 0xc013a8a220}, 0xc004cadd48?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:31 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x31abca0, 0xc013a8a220}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:36 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:837 +0x156\npanic({0x31abca0?, 0xc013a8a220?})\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/panic.go:770 +0x132\ncosmossdk.io/store/iavl.(*Store).Get(0xc000d82460?, {0xc016fad400?, 0xc013a8a1e0?, 0xc01cc9edc8?})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/iavl/store.go:202 +0xcb\ncosmossdk.io/store/cache.(*CommitKVStoreCache).Get(0xc001462d80, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cache/cache.go:111 +0xd1\ncosmossdk.io/store/cachekv.(*Store).Get(0xc024a297a0, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cachekv/store.go:61 +0x13a\ncosmossdk.io/store/cachekv.(*Store).Get(0xc02e1fdad0, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cachekv/store.go:61 +0x13a\ncosmossdk.io/store/gaskv.(*Store).Get(0xc01d6d4d80, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/gaskv/store.go:37 +0x6b\ngithub.com/cosmos/cosmos-sdk/runtime.coreKVStore.Get(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/runtime/store.go:57\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.getGrant({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/keeper.go:67 +0x95\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.DispatchActions({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/keeper.go:127 +0x33b\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.Exec({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/msg_server.go:120 +0x252\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler.func1({0x47dd778?, 0xc027595080?}, {0x36018a0?, 0xc023cfd5c0?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:449 +0xcb\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x47dc730, 0xc01d6d6e08}, {0xc004cb0170?, 0x412c25?}, 0x358?, 0xc0271b8960)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:175 +0x8a\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler({0x3711d80, 0xc0017dc7d0}, {0x47dc730, 0xc01d6d6e08}, 0x4033b38, 0xc014769e20)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:451 +0x143\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x47dcb20, 0x68ecb00}, {0x47fcd40, 0xc01d5c7340}, {{0x0, 0x0}, {0xc0012f7390, 0x7}, 0x16ec445, {0x1a63d7f9, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:198 +0x37f\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x47dcb20, 0x68ecb00}, {0x47fcd40, 0xc01d5c7340}, {{0x0, 0x0}, {0xc0012f7390, 0x7}, 0x16ec445, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:1010 +0x1e6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc00048fd48, 0x7, {0xc022a9ef00, 0x1d6, 0x1d6})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:948 +0x114c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).deliverTx(0xc00048fd48, {0xc022a9ef00?, 0x1d6?, 0xc01d210780?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:763 +0xc6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00048fd48, {0x47dcb20, 0x68ecb00}, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:790 +0x119c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00048fd48, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:884 +0x16e\ngithub.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/server/cmt_abci.go:44\ngithub.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0xc001bd5ca0?, {0x47dd858?, 0x68ecb00?}, 0x7efe1d2e9c08?)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:185 +0xcd\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc001032b70, {0x47dd858, 0x68ecb00}, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:104 +0x170\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc006b37f0a, 0x6}}, {0xc006b37f80, 0x7}, 0xf1b828, 0x16ec444, {{0xc02f003f20, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:213 +0x5c5\ngithub.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc0007592c0, 0x0)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:532 +0x1406\ngithub.com/cometbft/cometbft/blocksync.(*Reactor).OnStart.func1()\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:132 +0x54\ncreated by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 1\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:130 +0x8b\n: panic" module=server

Paloma and pigeon versions and logs

Section description

Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

v2.2.1

5:56PM INF executed block app_hash=91FBBD92B354EF66695899CEB5B9DA1EE668BDC349A4AA2A1F561F56ECB73670 height=24036417 module=state
5:56PM INF committed state block_app_hash=FBD4CEF7895DD215FBC6F4279F068BFF8DE68D35C5599A31D742D762DCBB32E6 height=24036417 module=state
5:56PM INF indexed block events height=24036417 module=txindex
5:56PM INF abci-validator-size module=x/palomaconsensus
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=arbitrum-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=base-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=blast-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=gnosis-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=matic-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=op-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to all chains module=x/evm smart-contract-id=5
5:56PM INF finalized block block_app_hash=4B4EA858DF748EC75B5582ECDC466F384D9C71104BDD412FBF1BF4C2BC4B3A9C height=24036418 module=state num_txs_res=16 num_val_updates=0
5:56PM INF executed block app_hash=4B4EA858DF748EC75B5582ECDC466F384D9C71104BDD412FBF1BF4C2BC4B3A9C height=24036418 module=state
5:56PM INF committed state block_app_hash=91FBBD92B354EF66695899CEB5B9DA1EE668BDC349A4AA2A1F561F56ECB73670 height=24036418 module=state
5:56PM INF indexed block events height=24036418 module=txindex
5:56PM INF added message signature. chain-reference-id=blast-main chain-type=evm message-id=302480 module=x/palomaconsensus queue-type-name=evm/blast-main/evm-turnstone-message signed-by-address=0xbefA0e4709A802C1dFc56F9d279cafE913CB866e validator=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg
5:56PM INF abci-validator-size module=x/palomaconsensus
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=arbitrum-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=base-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=blast-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=gnosis-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=matic-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=op-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to all chains module=x/evm smart-contract-id=5
5:56PM INF finalized block block_app_hash=B4D28D1E582F19AFA182538B872EB9F603555261FD104C94C20BE6E22451DBD5 height=24036419 module=state num_txs_res=9 num_val_updates=0
5:56PM INF executed block app_hash=B4D28D1E582F19AFA182538B872EB9F603555261FD104C94C20BE6E22451DBD5 height=24036419 module=state
5:56PM INF committed state block_app_hash=4B4EA858DF748EC75B5582ECDC466F384D9C71104BDD412FBF1BF4C2BC4B3A9C height=24036419 module=state
5:56PM INF indexed block events height=24036419 module=txindex
5:56PM INF abci-validator-size module=x/palomaconsensus
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=arbitrum-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=base-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=blast-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=gnosis-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=matic-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=op-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to all chains module=x/evm smart-contract-id=5
5:56PM INF finalized block block_app_hash=3CBC4D7F34EEAF9E9BE13BF266BDFB5DA708B9E6463FD840B1E7C9B6A2096883 height=24036420 module=state num_txs_res=10 num_val_updates=0
5:56PM INF executed block app_hash=3CBC4D7F34EEAF9E9BE13BF266BDFB5DA708B9E6463FD840B1E7C9B6A2096883 height=24036420 module=state
5:56PM INF committed state block_app_hash=B4D28D1E582F19AFA182538B872EB9F603555261FD104C94C20BE6E22451DBD5 height=24036420 module=state
5:56PM INF indexed block events height=24036420 module=txindex
5:56PM ERR panic recovered in runTx err="recovered: Value missing for key [0 0 0 0 1 110 14 115 0 0 0 1] corresponding to nodeKey 7300000000016e0e7300000001\nstack:\ngoroutine 301 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x31abca0, 0xc013a8a220})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:74 +0x25\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x31abca0?, 0xc013a8a220?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:42 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x31abca0, 0xc013a8a220}, 0xc004cadd48?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:31 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x31abca0, 0xc013a8a220}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:36 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:837 +0x156\npanic({0x31abca0?, 0xc013a8a220?})\n\t/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/panic.go:770 +0x132\ncosmossdk.io/store/iavl.(*Store).Get(0xc000d82460?, {0xc016fad400?, 0xc013a8a1e0?, 0xc01cc9edc8?})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/iavl/store.go:202 +0xcb\ncosmossdk.io/store/cache.(*CommitKVStoreCache).Get(0xc001462d80, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cache/cache.go:111 +0xd1\ncosmossdk.io/store/cachekv.(*Store).Get(0xc024a297a0, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cachekv/store.go:61 +0x13a\ncosmossdk.io/store/cachekv.(*Store).Get(0xc02e1fdad0, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/cachekv/store.go:61 +0x13a\ncosmossdk.io/store/gaskv.(*Store).Get(0xc01d6d4d80, {0xc016fad400, 0x4e, 0x4e})\n\t/home/runner/go/pkg/mod/cosmossdk.io/[email protected]/gaskv/store.go:37 +0x6b\ngithub.com/cosmos/cosmos-sdk/runtime.coreKVStore.Get(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/runtime/store.go:57\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.getGrant({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/keeper.go:67 +0x95\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.DispatchActions({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/keeper.go:127 +0x33b\ngithub.com/cosmos/cosmos-sdk/x/authz/keeper.Keeper.Exec({{0x479be60, 0xc0018c4328}, {0x4826c10, 0xc0016cd670}, {0x47ae4f8, 0xc00130fec0}, {0x47dc960, 0xc001d06000}, {0x47b8e00, 0xc001428a08}}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/keeper/msg_server.go:120 +0x252\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler.func1({0x47dd778?, 0xc027595080?}, {0x36018a0?, 0xc023cfd5c0?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:449 +0xcb\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x47dc730, 0xc01d6d6e08}, {0xc004cb0170?, 0x412c25?}, 0x358?, 0xc0271b8960)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:175 +0x8a\ngithub.com/cosmos/cosmos-sdk/x/authz._Msg_Exec_Handler({0x3711d80, 0xc0017dc7d0}, {0x47dc730, 0xc01d6d6e08}, 0x4033b38, 0xc014769e20)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/authz/tx.pb.go:451 +0x143\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x47dcb20, 0x68ecb00}, {0x47fcd40, 0xc01d5c7340}, {{0x0, 0x0}, {0xc0012f7390, 0x7}, 0x16ec445, {0x1a63d7f9, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/msg_service_router.go:198 +0x37f\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x47dcb20, 0x68ecb00}, {0x47fcd40, 0xc01d5c7340}, {{0x0, 0x0}, {0xc0012f7390, 0x7}, 0x16ec445, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:1010 +0x1e6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc00048fd48, 0x7, {0xc022a9ef00, 0x1d6, 0x1d6})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:948 +0x114c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).deliverTx(0xc00048fd48, {0xc022a9ef00?, 0x1d6?, 0xc01d210780?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/baseapp.go:763 +0xc6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00048fd48, {0x47dcb20, 0x68ecb00}, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:790 +0x119c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00048fd48, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:884 +0x16e\ngithub.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/[email protected]/server/cmt_abci.go:44\ngithub.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0xc001bd5ca0?, {0x47dd858?, 0x68ecb00?}, 0x7efe1d2e9c08?)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:185 +0xcd\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc001032b70, {0x47dd858, 0x68ecb00}, 0xc0132e7680)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:104 +0x170\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc006b37f0a, 0x6}}, {0xc006b37f80, 0x7}, 0xf1b828, 0x16ec444, {{0xc02f003f20, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:213 +0x5c5\ngithub.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc0007592c0, 0x0)\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:532 +0x1406\ngithub.com/cometbft/cometbft/blocksync.(*Reactor).OnStart.func1()\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:132 +0x54\ncreated by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 1\n\t/home/runner/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:130 +0x8b\n: panic" module=server
5:56PM INF abci-validator-size module=x/palomaconsensus
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=arbitrum-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=base-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=blast-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=gnosis-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=matic-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to EVM chain chain-reference-id=op-main module=x/evm smart-contract-id=5
5:56PM INF trying to deploy smart contract to all chains module=x/evm smart-contract-id=5
5:56PM INF finalized block block_app_hash=1B68F5E437A29C9BC3E3B7F3CC89F78AE5CAF170C6BEAD222C4A3A127D82F23D height=24036421 module=state num_txs_res=14 num_val_updates=0
5:56PM INF executed block app_hash=1B68F5E437A29C9BC3E3B7F3CC89F78AE5CAF170C6BEAD222C4A3A127D82F23D height=24036421 module=state
5:56PM INF committed state block_app_hash=3CBC4D7F34EEAF9E9BE13BF266BDFB5DA708B9E6463FD840B1E7C9B6A2096883 height=24036421 module=state
5:56PM INF indexed block events height=24036421 module=txindex
5:56PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected 1B68F5E437A29C9BC3E3B7F3CC89F78AE5CAF170C6BEAD222C4A3A127D82F23D, got 3D1006E73D0C7C1452441A38F8369D53028C97AC9A52EE058E3CE2260A22C814" module=blocksync
5:56PM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.AppHash.  Expected 1B68F5E437A29C9BC3E3B7F3CC89F78AE5CAF170C6BEAD222C4A3A127D82F23D, got 3D1006E73D0C7C1452441A38F8369D53028C97AC9A52EE058E3CE2260A22C814" module=p2p peer="Peer{MConn{79.143.179.196:26656} 2c6772b11c1f9eff2a923eb2bf808543cdd501c5 out}"

[taariq]

In the https://paloma.explorers.guru/block/24036421 block, I see there's a delegation tx https://paloma.explorers.guru/transaction/2897576E27E693DDB30F2427D9AAEDE1B3FACB3ABB1004BA57D5140A1E78F048?height=24036421 from a Paloma Lightnode. it may be related.

[taariq]

We chain halted today after upgrade.

Sep 22 19:32:16 host palomad[2652531]: 7:32PM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash. Expected 6F2516BE4832BF4777554CF895D447C181FBB709C3470576F0ED3D443F7E2208, got 7DFE59223CF548C5661A194AF2B78E7163E50FE58A4348811D91BD9B0010E483" height=24160804 module=consensus round=0

Halted block: https://paloma.explorers.guru/block/24160803

[byte-bandit]

This is where the panic is coming from. Given that it panics all the way down in the store implementation, I don't really understand why they're catching an error here and also try to handle the case of value not found.

But ultimately it means this grant wasn't found when expected to be there. @maharifu You're more involved with the authz module, I imagine we can assume the grant exists at this point, seeing as there's already a TX that contains a dispatch call, and other nodes didn't panic?

If so, this is the third occurrence we've come across of the IAVL implementation not persisting files accordingly in some cases. We've been in touch with the Cosmos team, but they didn't really have an explanation for this either.

We're looking to upgrade to v0.50.9, which might contain bug fixes to address this issue.

[maharifu]

For the specific transaction on that block, the grant should be there:

grants:
- authorization:
    type: cosmos-sdk/GenericAuthorization
    value:
      msg: /cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward
  grantee: paloma1gh3c0mqrrc40057vtydxhy8pk27a5yhffpsh23
  granter: paloma182ettr33r6cjh5sly58rl75wzcw8dn65pmey6x
- authorization:
    type: cosmos-sdk/GenericAuthorization
    value:
      msg: /cosmos.staking.v1beta1.MsgDelegate
  grantee: paloma1gh3c0mqrrc40057vtydxhy8pk27a5yhffpsh23
  granter: paloma182ettr33r6cjh5sly58rl75wzcw8dn65pmey6x
pagination:
  total: "2"

I agree this seems related to a previous issue and we should upgrade.

[taariq]

Sorry. Wrong ticket!