Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MBA - New install stops syncing on block 1131146 #2357

Closed
JB273 opened this issue Jun 17, 2023 · 6 comments
Closed

MBA - New install stops syncing on block 1131146 #2357

JB273 opened this issue Jun 17, 2023 · 6 comments

Comments

@JB273
Copy link

JB273 commented Jun 17, 2023

Installed a new server with Ubuntu 22.04 to setup a new MBA Archive RPC node.
It starts of fine, but after a while it stalls syncing blocks, currently it stalled at block 1131146.
Restarting the service gives the following issues in the log:
Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] panicked at 'Digest item must match that calculated.', /home/builder/cargo/git/checkouts/substrate-189071a041b0d328/ecd04ec/frame/executive/src/lib.rs:668:13 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachableinstruction executed Jun 17 13:43:16 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:16 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:16 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:16 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] 💔 Error importing block 0x236e62a41257327fed36be064504a0f765169c0e8c50328033394f6b5ae45c16: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed Jun 17 13:43:16 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:16 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:16 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:16 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block
and
Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Skipping chain: unable to gather info about imported block (0xbcebf6cdd7959a4ef3db86c8abfbaf4aea0ff1bcb4571106691d11d97ac33e52, 10782350): session info unavailable Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:19 rpc05-var moonbase[50064]: 2023-06-17 13:43:19 [Relaychain] 💤 Idle (4 peers), best: #10782352 (0x4d32…eb2c), finalized #10782349 (0x4375…d5d9), ⬇ 160.4kiB/s ⬆ 79.8kiB/s Jun 17 13:43:19 rpc05-var moonbase[50064]: 2023-06-17 13:43:19 [🌗] ⚙️ Syncing, target=#4559211 (1 peers), best: #1131146 (0xe3f7…8c36), finalized #3497 (0x6331…793a), ⬇ 5.0MiB/s ⬆ 38.5kiB/s Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] panicked at 'Digest item must match that calculated.', /home/builder/cargo/git/checkouts/substrate-189071a041b0d328/ecd04ec/frame/executive/src/lib.rs:668:13 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachableinstruction executed Jun 17 13:43:21 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:21 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:21 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:21 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] 💔 Error importing block 0x236e62a41257327fed36be064504a0f765169c0e8c50328033394f6b5ae45c16: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed Jun 17 13:43:21 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:21 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:21 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:21 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block

@JB273
Copy link
Author

JB273 commented Jun 20, 2023

Upgraded binary to v0.32.1, started a clean sync (Archival) and is stalls on the same block number. (1131146)

@crystalin
Copy link
Collaborator

Thank you, we are taking a look at it now

@JB273
Copy link
Author

JB273 commented Jun 20, 2023

ExecStart=/home/pops/moonbase/moonbase \
     --port 30335 \
     --ws-port 9946 \
     --ws-max-connections 10000 \
     --execution wasm \
     --wasm-execution compiled \
     --state-pruning=archive \
     --prometheus-external \
     --prometheus-port 9617 \
     --trie-cache-size 4 \
     --db-cache 32000 \
     --base-path /home/pops/moonbase \
     --ethapi=debug,trace,txpool \
     --wasm-runtime-overrides=/home/pops/moonbase/wasm \
     --chain alphanet \
     --name "P-OPS Team Moonbase RPC-2" \
     --runtime-cache-size 64 \
     --rpc-max-response-size 128 \
     --unsafe-rpc-external \
     --unsafe-ws-external \
     --rpc-cors all \
     -- \
     --port 30336 \
     --ws-port 9947 \
     --execution wasm \
     --pruning=1000 \
     --prometheus-external \
     --prometheus-port 9618 \
     --name="P-OPS Team Moonbase RPC-2 (Embedded Relay)"

@crystalin
Copy link
Collaborator

Multiple engineers, including me have been trying to reproduce this, but none of us got issue synchronizing past 1131147.
I'm not sure what happens in your case, but I suspect that the node might have been turned off in a middle of a DB process leaving data unstable when you restarted it, leading to the node not able to process that block anymore.

Can you resync from scratch, using client v0.32.1 to see if this happens again ?

@JB273
Copy link
Author

JB273 commented Jun 21, 2023

Resynced from scratch... still not passing block 1131146, not sure why this is... It's also odd as on the same machine Moonbeam synced flawless!

@RomarQ
Copy link
Contributor

RomarQ commented Nov 5, 2024

Stale issue, closing.

@RomarQ RomarQ closed this as completed Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants