Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Acala mainnet stuck on finalized block #5446885 #2774

Closed
andreclaro opened this issue Jun 19, 2024 · 7 comments
Closed

Acala mainnet stuck on finalized block #5446885 #2774

andreclaro opened this issue Jun 19, 2024 · 7 comments

Comments

@andreclaro
Copy link

Describe the bug
A new acala mainnet node, which has been syncing from genesis, is stuck on finalized block #5446885

Expected Behavior
Sync from genesis till the latest block without issues.

Current Behavior
Suck on finalized block #5446885.

Steps to Reproduce

  1. Start acala client without database
  2. Wait a few days

Additional context

  • Acala version: acala 2.25.0-eb961734875
  • OS: Ubuntu 22.04.3 LTS
  • Command line options:
/usr/local/bin/acala \
    --base-path=/var/lib/acala/data \
    --chain=acala \
    --name=xLabs \
# Pruning Acala DB
# Note: paritydb database is recommend when pruning is enabled
    --database=paritydb \
    --state-pruning=200000 \
    --blocks-pruning=200000 \
    --rpc-cors=all \
    --rpc-external \
    --rpc-port=9933 \
    --rpc-max-connections=500 \
    --port=30333 \
    --prometheus-external \
    --prometheus-port=9615 \
    -- \
# Pruning Relaychain DB
    --database=paritydb \
    --state-pruning=200000 \
    --blocks-pruning=200000
  • Logs:
Jun 19 13:35:30 m-acala-02 acala[3326290]: 2024-06-19 13:35:30 ✨ Imported #6417394 (0xc2f3…180b)
Jun 19 13:35:35 m-acala-02 acala[3326290]: 2024-06-19 13:35:35 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 6.7kiB/s ⬆ 8.4kiB/s
Jun 19 13:35:35 m-acala-02 acala[3326290]: 2024-06-19 13:35:35 💤 Idle (9 peers), best: #6417394 (0xc2f3…180b), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.6kiB/s
Jun 19 13:35:40 m-acala-02 acala[3326290]: 2024-06-19 13:35:40 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 5.0kiB/s ⬆ 8.4kiB/s
Jun 19 13:35:40 m-acala-02 acala[3326290]: 2024-06-19 13:35:40 💤 Idle (9 peers), best: #6417394 (0xc2f3…180b), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 ✨ Imported #6417395 (0x0040…92bf)
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 13:35:42 m-acala-02 acala[3326290]: 2024-06-19 13:35:42 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -1024 to -2147483648. Reason: same small block request multiple times. Banned, disconnecting.
Jun 19 13:35:45 m-acala-02 acala[3326290]: 2024-06-19 13:35:45 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4339712 (0xe134…12da), finalized #4339712 (0xe134…12da), ⬇ 5.0kiB/s ⬆ 7.4kiB/s
Jun 19 13:35:45 m-acala-02 acala[3326290]: 2024-06-19 13:35:45 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 7.5kiB/s ⬆ 254.7kiB/s
Jun 19 13:35:50 m-acala-02 acala[3326290]: 2024-06-19 13:35:50 [Relaychain] ⚙️  Syncing 153.6 bps, target=#21283499 (40 peers), best: #4340480 (0x736b…c762), finalized #4340224 (0x6def…3e37), ⬇ 81.4kiB/s ⬆ 5.8kiB/s
Jun 19 13:35:50 m-acala-02 acala[3326290]: 2024-06-19 13:35:50 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 [Relaychain] ⚙️  Syncing 395.4 bps, target=#21283499 (40 peers), best: #4342457 (0xb309…349b), finalized #4342273 (0x3dd4…c76b), ⬇ 669.1kiB/s ⬆ 5.1kiB/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 💤 Idle (9 peers), best: #6417395 (0x0040…92bf), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 57 B/s
Jun 19 13:35:55 m-acala-02 acala[3326290]: 2024-06-19 13:35:55 ✨ Imported #6417396 (0x046b…375e)
Jun 19 13:36:00 m-acala-02 acala[3326290]: 2024-06-19 13:36:00 [Relaychain] ⚙️  Syncing 444.0 bps, target=#21283499 (40 peers), best: #4344677 (0x755b…f84d), finalized #4344320 (0x35fa…353c), ⬇ 241.2kiB/s ⬆ 2.3kiB/s
Jun 19 13:36:00 m-acala-02 acala[3326290]: 2024-06-19 13:36:00 💤 Idle (9 peers), best: #6417396 (0x046b…375e), finalized #5446885 (0x1f31…4d80), ⬇ 11.0kiB/s ⬆ 21.9kiB/s
Jun 19 13:36:05 m-acala-02 acala[3326290]: 2024-06-19 13:36:05 [Relaychain] ⚙️  Syncing 375.6 bps, target=#21283499 (40 peers), best: #4346555 (0xe369…a917), finalized #4346368 (0x5f87…c70a), ⬇ 189.9kiB/s ⬆ 3.3kiB/s
Jun 19 13:36:05 m-acala-02 acala[3326290]: 2024-06-19 13:36:05 💤 Idle (9 peers), best: #6417396 (0x046b…375e), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:36:06 m-acala-02 acala[3326290]: 2024-06-19 13:36:06 ✨ Imported #6417397 (0xe5cd…04c4)
Jun 19 13:36:10 m-acala-02 acala[3326290]: 2024-06-19 13:36:10 [Relaychain] ⚙️  Syncing 474.6 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 225.4kiB/s ⬆ 2.1kiB/s
Jun 19 13:36:10 m-acala-02 acala[3326290]: 2024-06-19 13:36:10 💤 Idle (9 peers), best: #6417397 (0xe5cd…04c4), finalized #5446885 (0x1f31…4d80), ⬇ 18.2kiB/s ⬆ 7.5kiB/s
Jun 19 13:36:15 m-acala-02 acala[3326290]: 2024-06-19 13:36:15 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 2.9kiB/s ⬆ 0.9kiB/s
Jun 19 13:36:15 m-acala-02 acala[3326290]: 2024-06-19 13:36:15 💤 Idle (9 peers), best: #6417397 (0xe5cd…04c4), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:36:18 m-acala-02 acala[3326290]: 2024-06-19 13:36:18 ✨ Imported #6417398 (0x58ca…7c9a)
Jun 19 13:36:20 m-acala-02 acala[3326290]: 2024-06-19 13:36:20 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 2.2kiB/s ⬆ 1.0kiB/s
Jun 19 13:36:20 m-acala-02 acala[3326290]: 2024-06-19 13:36:20 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 4.0kiB/s ⬆ 25.4kiB/s
Jun 19 13:36:25 m-acala-02 acala[3326290]: 2024-06-19 13:36:25 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 3.1kiB/s ⬆ 1.2kiB/s
Jun 19 13:36:25 m-acala-02 acala[3326290]: 2024-06-19 13:36:25 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 44 B/s ⬆ 44 B/s
Jun 19 13:36:27 m-acala-02 acala[3326290]: 2024-06-19 13:36:27 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4348928 (0xf168…c5a4), finalized #4348928 (0xf168…c5a4), ⬇ 138.3kiB/s ⬆ 77.0kiB/s
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 💤 Idle (9 peers), best: #6417398 (0x58ca…7c9a), finalized #5446885 (0x1f31…4d80), ⬇ 1.5kiB/s ⬆ 0.8kiB/s
Jun 19 13:36:30 m-acala-02 acala[3326290]: 2024-06-19 13:36:30 ✨ Imported #6417399 (0x6d82…ec4b)
Jun 19 13:36:35 m-acala-02 acala[3326290]: 2024-06-19 13:36:35 [Relaychain] ⚙️  Syncing 469.0 bps, target=#21283499 (40 peers), best: #4351273 (0xf916…edea), finalized #4350976 (0x723c…71e0), ⬇ 224.3kiB/s ⬆ 10.5kiB/s
Jun 19 13:36:35 m-acala-02 acala[3326290]: 2024-06-19 13:36:35 💤 Idle (9 peers), best: #6417399 (0x6d82…ec4b), finalized #5446885 (0x1f31…4d80), ⬇ 8.0kiB/s ⬆ 8.0kiB/s
Jun 19 13:36:37 m-acala-02 acala[3326290]: 2024-06-19 13:36:37 [Relaychain] 💔 Ignored block (#21285723 -- 0x6360…8183) announcement from 12D3KooWJMffzdMYBu8PBd6CYnSyjwMNoFxBkNQGwrCbpkgp3FZm because all validation slots for this peer are occupied.
Jun 19 13:36:37 m-acala-02 acala[3326290]: 2024-06-19 13:36:37 [Relaychain] 💔 Ignored block (#21285724 -- 0xa8f1…b20c) announcement from 12D3KooWJMffzdMYBu8PBd6CYnSyjwMNoFxBkNQGwrCbpkgp3FZm because all validation slots for this peer are occupied.
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 [Relaychain] ⚙️  Syncing 514.8 bps, target=#21283499 (40 peers), best: #4353847 (0x1a91…53c5), finalized #4353536 (0x4534…ee69), ⬇ 526.7kiB/s ⬆ 133.5kiB/s
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 💤 Idle (9 peers), best: #6417399 (0x6d82…ec4b), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 51 B/s
Jun 19 13:36:40 m-acala-02 acala[3326290]: 2024-06-19 13:36:40 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:36:42 m-acala-02 acala[3326290]: 2024-06-19 13:36:42 ✨ Imported #6417400 (0x1d7a…c4f8)
Jun 19 13:36:45 m-acala-02 acala[3326290]: 2024-06-19 13:36:45 [Relaychain] ⚙️  Syncing 245.0 bps, target=#21283499 (40 peers), best: #4355072 (0x3c86…3229), finalized #4355072 (0x3c86…3229), ⬇ 180.4kiB/s ⬆ 31.5kiB/s
Jun 19 13:36:45 m-acala-02 acala[3326290]: 2024-06-19 13:36:45 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 11.5kiB/s ⬆ 7.9kiB/s
Jun 19 13:36:49 m-acala-02 acala[3326290]: 2024-06-19 13:36:49 ✨ Imported #6417400 (0x7d55…3150)
Jun 19 13:36:50 m-acala-02 acala[3326290]: 2024-06-19 13:36:50 [Relaychain] ⚙️  Syncing 340.4 bps, target=#21283499 (40 peers), best: #4356774 (0x21c1…64e7), finalized #4356608 (0xb7b7…7d01), ⬇ 348.5kiB/s ⬆ 85.0kiB/s
Jun 19 13:36:50 m-acala-02 acala[3326290]: 2024-06-19 13:36:50 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 31.0kiB/s ⬆ 10.8kiB/s
Jun 19 13:36:55 m-acala-02 acala[3326290]: 2024-06-19 13:36:55 [Relaychain] ⚙️  Syncing 274.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 185.2kiB/s ⬆ 32.0kiB/s
Jun 19 13:36:55 m-acala-02 acala[3326290]: 2024-06-19 13:36:55 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 163.7kiB/s ⬆ 95.7kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 💤 Idle (9 peers), best: #6417400 (0x1d7a…c4f8), finalized #5446885 (0x1f31…4d80), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 ♻️  Reorg on #6417400,0x1d7a…c4f8 to #6417401,0xcf8c…2ac5, common ancestor #6417399,0x6d82…ec4b
Jun 19 13:37:00 m-acala-02 acala[3326290]: 2024-06-19 13:37:00 ✨ Imported #6417401 (0xcf8c…2ac5)
Jun 19 13:37:05 m-acala-02 acala[3326290]: 2024-06-19 13:37:05 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 73.8kiB/s ⬆ 35.9kiB/s
Jun 19 13:37:05 m-acala-02 acala[3326290]: 2024-06-19 13:37:05 💤 Idle (9 peers), best: #6417401 (0xcf8c…2ac5), finalized #5446885 (0x1f31…4d80), ⬇ 4.3kiB/s ⬆ 39.7kiB/s
Jun 19 13:37:10 m-acala-02 acala[3326290]: 2024-06-19 13:37:10 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 153.1kiB/s ⬆ 68.7kiB/s
Jun 19 13:37:10 m-acala-02 acala[3326290]: 2024-06-19 13:37:10 💤 Idle (9 peers), best: #6417401 (0xcf8c…2ac5), finalized #5446885 (0x1f31…4d80), ⬇ 0.1kiB/s ⬆ 70 B/s
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285719 -- 0x0aa7…72a2) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285720 -- 0x9c47…fb60) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285721 -- 0xf39e…48f3) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285721 -- 0x4304…351b) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285724 -- 0xa8f1…b20c) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285725 -- 0x6940…d82d) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:11 m-acala-02 acala[3326290]: 2024-06-19 13:37:11 [Relaychain] 💔 Ignored block (#21285726 -- 0x9b12…437c) announcement from 12D3KooWAPgvemiDBZekp7uKA4NNjp9XyMEtf1Fz6QDnMmc7NLWZ because all validation slots for this peer are occupied.
Jun 19 13:37:13 m-acala-02 acala[3326290]: 2024-06-19 13:37:13 ✨ Imported #6417402 (0x2cf2…0992)
Jun 19 13:37:15 m-acala-02 acala[3326290]: 2024-06-19 13:37:15 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 121.7kiB/s ⬆ 65.4kiB/s
Jun 19 13:37:15 m-acala-02 acala[3326290]: 2024-06-19 13:37:15 💤 Idle (9 peers), best: #6417402 (0x2cf2…0992), finalized #5446885 (0x1f31…4d80), ⬇ 7.7kiB/s ⬆ 22.1kiB/s
Jun 19 13:37:20 m-acala-02 acala[3326290]: 2024-06-19 13:37:20 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4358144 (0x437f…5eaf), finalized #4358144 (0x437f…5eaf), ⬇ 76.7kiB/s ⬆ 52.1kiB/s
Jun 19 13:37:20 m-acala-02 acala[3326290]: 2024-06-19 13:37:20 💤 Idle (9 peers), best: #6417402 (0x2cf2…0992), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:24 m-acala-02 acala[3326290]: 2024-06-19 13:37:24 ✨ Imported #6417403 (0xbd5a…8d09)
Jun 19 13:37:25 m-acala-02 acala[3326290]: 2024-06-19 13:37:25 [Relaychain] ⚙️  Syncing 407.8 bps, target=#21283499 (40 peers), best: #4360183 (0x06f8…4977), finalized #4359680 (0xa8c2…291c), ⬇ 353.5kiB/s ⬆ 86.7kiB/s
Jun 19 13:37:25 m-acala-02 acala[3326290]: 2024-06-19 13:37:25 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.6kiB/s
Jun 19 13:37:30 m-acala-02 acala[3326290]: 2024-06-19 13:37:30 [Relaychain] ⚙️  Syncing 478.4 bps, target=#21283499 (40 peers), best: #4362575 (0x5e61…3e97), finalized #4362240 (0xceeb…ccee), ⬇ 292.8kiB/s ⬆ 42.4kiB/s
Jun 19 13:37:30 m-acala-02 acala[3326290]: 2024-06-19 13:37:30 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:37:35 m-acala-02 acala[3326290]: 2024-06-19 13:37:35 [Relaychain] ⚙️  Syncing 253.2 bps, target=#21283499 (40 peers), best: #4363841 (0x8b11…f7b2), finalized #4363776 (0x408a…c4ae), ⬇ 262.4kiB/s ⬆ 78.0kiB/s
Jun 19 13:37:35 m-acala-02 acala[3326290]: 2024-06-19 13:37:35 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:36 m-acala-02 acala[3326290]: 2024-06-19 13:37:36 ✨ Imported #6417403 (0x0c0c…6b39)
Jun 19 13:37:40 m-acala-02 acala[3326290]: 2024-06-19 13:37:40 [Relaychain] ⚙️  Syncing 331.8 bps, target=#21283499 (40 peers), best: #4365500 (0xfdb6…8396), finalized #4365312 (0x0242…16c5), ⬇ 201.7kiB/s ⬆ 38.4kiB/s
Jun 19 13:37:40 m-acala-02 acala[3326290]: 2024-06-19 13:37:40 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 4.2kiB/s ⬆ 18.5kiB/s
Jun 19 13:37:41 m-acala-02 acala[3326290]: 2024-06-19 13:37:41 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:37:45 m-acala-02 acala[3326290]: 2024-06-19 13:37:45 [Relaychain] ⚙️  Syncing 558.6 bps, target=#21283499 (40 peers), best: #4368293 (0x0938…608d), finalized #4367872 (0x94de…ddf6), ⬇ 494.3kiB/s ⬆ 122.0kiB/s
Jun 19 13:37:45 m-acala-02 acala[3326290]: 2024-06-19 13:37:45 💤 Idle (9 peers), best: #6417403 (0xbd5a…8d09), finalized #5446885 (0x1f31…4d80), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
Jun 19 13:37:48 m-acala-02 acala[3326290]: 2024-06-19 13:37:48 ♻️  Reorg on #6417403,0xbd5a…8d09 to #6417404,0x844e…d5e4, common ancestor #6417402,0x2cf2…0992
Jun 19 13:37:48 m-acala-02 acala[3326290]: 2024-06-19 13:37:48 ✨ Imported #6417404 (0x844e…d5e4)
Jun 19 13:37:50 m-acala-02 acala[3326290]: 2024-06-19 13:37:50 [Relaychain] ⚙️  Syncing 120.6 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 120.7kiB/s ⬆ 46.3kiB/s
Jun 19 13:37:50 m-acala-02 acala[3326290]: 2024-06-19 13:37:50 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 7.9kiB/s ⬆ 11.4kiB/s
Jun 19 13:37:55 m-acala-02 acala[3326290]: 2024-06-19 13:37:55 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 162.3kiB/s ⬆ 79.3kiB/s
Jun 19 13:37:55 m-acala-02 acala[3326290]: 2024-06-19 13:37:55 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:38:00 m-acala-02 acala[3326290]: 2024-06-19 13:38:00 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4368896 (0x4d9f…a405), finalized #4368896 (0x4d9f…a405), ⬇ 25.7kiB/s ⬆ 21.3kiB/s
Jun 19 13:38:00 m-acala-02 acala[3326290]: 2024-06-19 13:38:00 💤 Idle (9 peers), best: #6417404 (0x844e…d5e4), finalized #5446885 (0x1f31…4d80), ⬇ 38 B/s ⬆ 38 B/s
Jun 19 13:38:01 m-acala-02 acala[3326290]: 2024-06-19 13:38:01 ✨ Imported #6417405 (0xe20f…3572)
Jun 19 13:38:05 m-acala-02 acala[3326290]: 2024-06-19 13:38:05 [Relaychain] ⚙️  Syncing 171.4 bps, target=#21283499 (40 peers), best: #4369753 (0xb78e…8959), finalized #4369408 (0xc265…54ba), ⬇ 245.2kiB/s ⬆ 79.9kiB/s
Jun 19 13:38:05 m-acala-02 acala[3326290]: 2024-06-19 13:38:05 💤 Idle (9 peers), best: #6417405 (0xe20f…3572), finalized #5446885 (0x1f31…4d80), ⬇ 11.4kiB/s ⬆ 7.9kiB/s
Jun 19 13:38:10 m-acala-02 acala[3326290]: 2024-06-19 13:38:10 [Relaychain] ⚙️  Syncing 341.4 bps, target=#21283499 (40 peers), best: #4371460 (0x9e0f…7803), finalized #4371456 (0x4c13…10de), ⬇ 259.3kiB/s ⬆ 32.9kiB/s
Jun 19 13:38:10 m-acala-02 acala[3326290]: 2024-06-19 13:38:10 💤 Idle (9 peers), best: #6417405 (0xe20f…3572), finalized #5446885 (0x1f31…4d80), ⬇ 64 B/s ⬆ 64 B/s
Jun 19 13:38:12 m-acala-02 acala[3326290]: 2024-06-19 13:38:12 ✨ Imported #6417406 (0x5527…76fe)
Jun 19 13:38:15 m-acala-02 acala[3326290]: 2024-06-19 13:38:15 [Relaychain] ⚙️  Syncing 445.4 bps, target=#21283499 (40 peers), best: #4373687 (0xa40a…3b2e), finalized #4373504 (0x24a3…9229), ⬇ 335.4kiB/s ⬆ 59.1kiB/s
Jun 19 13:38:15 m-acala-02 acala[3326290]: 2024-06-19 13:38:15 💤 Idle (9 peers), best: #6417406 (0x5527…76fe), finalized #5446885 (0x1f31…4d80), ⬇ 21.7kiB/s ⬆ 7.5kiB/s
Jun 19 13:38:20 m-acala-02 acala[3326290]: 2024-06-19 13:38:20 [Relaychain] ⚙️  Syncing 488.8 bps, target=#21283499 (40 peers), best: #4376136 (0x3ca5…5ad5), finalized #4376064 (0xdd54…c925), ⬇ 271.8kiB/s ⬆ 24.9kiB/s
Jun 19 13:38:20 m-acala-02 acala[3326290]: 2024-06-19 13:38:20 💤 Idle (9 peers), best: #6417406 (0x5527…76fe), finalized #5446885 (0x1f31…4d80), ⬇ 25 B/s ⬆ 25 B/s
Jun 19 13:38:24 m-acala-02 acala[3326290]: 2024-06-19 13:38:24 ✨ Imported #6417407 (0xb3fa…79b2)
Jun 19 13:38:25 m-acala-02 acala[3326290]: 2024-06-19 13:38:25 [Relaychain] ⚙️  Syncing 395.2 bps, target=#21283499 (40 peers), best: #4378112 (0x4d81…be77), finalized #4378112 (0x4d81…be77), ⬇ 312.5kiB/s ⬆ 56.0kiB/s
Jun 19 13:38:25 m-acala-02 acala[3326290]: 2024-06-19 13:38:25 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 3.9kiB/s ⬆ 28.9kiB/s
Jun 19 13:38:27 m-acala-02 acala[3326290]: 2024-06-19 13:38:27 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:38:30 m-acala-02 acala[3326290]: 2024-06-19 13:38:30 [Relaychain] ⚙️  Syncing 289.0 bps, target=#21283499 (40 peers), best: #4379557 (0xf518…c896), finalized #4379136 (0x28d0…864b), ⬇ 192.9kiB/s ⬆ 47.6kiB/s
Jun 19 13:38:30 m-acala-02 acala[3326290]: 2024-06-19 13:38:30 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 5.1kiB/s ⬆ 2.2kiB/s
Jun 19 13:38:32 m-acala-02 acala[3326290]: 2024-06-19 13:38:32 [Relaychain] Sending fatal alert BadCertificate
Jun 19 13:38:35 m-acala-02 acala[3326290]: 2024-06-19 13:38:35 [Relaychain] ⚙️  Syncing 427.8 bps, target=#21283499 (40 peers), best: #4381696 (0x0d04…3e90), finalized #4381696 (0x0d04…3e90), ⬇ 356.4kiB/s ⬆ 73.0kiB/s
Jun 19 13:38:35 m-acala-02 acala[3326290]: 2024-06-19 13:38:35 💤 Idle (9 peers), best: #6417407 (0xb3fa…79b2), finalized #5446885 (0x1f31…4d80), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
Jun 19 13:38:37 m-acala-02 acala[3326290]: 2024-06-19 13:38:37 ✨ Imported #6417408 (0xd7c9…528c)
Jun 19 13:38:40 m-acala-02 acala[3326290]: 2024-06-19 13:38:40 [Relaychain] ⚙️  Syncing 409.6 bps, target=#21283499 (40 peers), best: #4383744 (0x569e…c117), finalized #4383744 (0x569e…c117), ⬇ 325.3kiB/s ⬆ 54.9kiB/s
Jun 19 13:38:40 m-acala-02 acala[3326290]: 2024-06-19 13:38:40 💤 Idle (9 peers), best: #6417408 (0xd7c9…528c), finalized #5446885 (0x1f31…4d80), ⬇ 12.6kiB/s ⬆ 22.5kiB/s
Jun 19 13:38:45 m-acala-02 acala[3326290]: 2024-06-19 13:38:45 [Relaychain] ⚙️  Syncing  0.0 bps, target=#21283499 (40 peers), best: #4383744 (0x569e…c117), finalized #4383744 (0x569e…c117), ⬇ 91.4kiB/s ⬆ 45.3kiB/s
Jun 19 13:38:45 m-acala-02 acala[3326290]: 2024-06-19 13:38:45 💤 Idle (9 peers), best: #6417408 (0xd7c9…528c), finalized #5446885 (0x1f31…4d80), ⬇ 3.5kiB/s ⬆ 1.9kiB/s
@andreclaro
Copy link
Author

last days - block height
image

last 12 hours - block height
image

@andreclaro
Copy link
Author

I see a few message like Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.

Example:

Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -1024 to -2147483648. Reason: same small block request multiple times. Banned, disconnecting.
Jun 19 11:00:17 m-acala-02 acala[3326290]: 2024-06-19 11:00:17 Report 12D3KooWJW6iSkaD6DYV8h8V5pzzVdBsS6rHSpN8yL7toKPxrkDa: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.

@eskimor
Copy link

eskimor commented Jun 20, 2024

Does a restart help?

@andreclaro
Copy link
Author

No, it didn't help. I tried multiple times to restart the node.

Now, I deleted the acala database and the node is syncing from genesis again... I will update this issue...

image

@eskimor
Copy link

eskimor commented Jun 20, 2024

Do you still have the database around? Restarting with debug or even trace logs might reveal what is going on.

RUST_LOG=debug /your/collator/binary --args-you-normally-pass

Trace for everything is likely to spammy, but only for sync should work:

RUST_LOG=sync=trace /your/collator/binary --args-you-normally-pass

@andreclaro
Copy link
Author

I will enable that level of logging if I see this or any other issues. Thanks

@andreclaro
Copy link
Author

I didn't see this issue again. Closing for now.

@andreclaro andreclaro closed this as not planned Won't fix, can't repro, duplicate, stale Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eskimor @andreclaro