blutgang doesn't finalize blocks with reth and nethermind

[nidhi-singh02]

Describe the bug
I am running a custom network with multiple EL clients and a CL. Blutgang works with geth alone, but doesn't work with reth and nethermind even solo.

To Reproduce
Steps to reproduce the behavior:
Run a network with EL and CL clients and add blutgang as load balancer for JSON RPC endpoints.

Expected behavior
My blocks are not getting finalised via testing when using solo reth and solo nethermind.

Specs:

• OS: Running in docker containers via kurtosis
• Kernel: [e.g. Linux 6.2.0-27-generic]
• Blutgang version: Used docker image both makemake1337/blutgang:latest and makemake1337/blutgang:master
• Kurtosis version
  Blutgang options:
  Kept all the blutgang options by default as in kurtosis ethereum package. This is how config toml template looks like https://github.com/kurtosis-tech/ethereum-package/blob/main/static_files/blutgang-config/config.toml.tmpl
• DB mode: HighThroughput
• DB compression: no
• Cache capacity: 1000000000
• MA length: 100
• RPCs used: reth and nethermind

Additional context
I also tried playing around with some parameters of toml file such as setting readonly flag in admin section to false and increasing the expected_block_time , health_check_ttl and ttl, but no luck.

[makemake-kbo]

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients.

Try the 0.3.5-debug-verbose docker image.

It should have more logging outputs that should help us debug this.

[nidhi-singh02]

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients.

Try the 0.3.5-debug-verbose docker image.

It should have more logging outputs that should help us debug this.

Sure, Let me run using this docker image. Thank you for pushing the image :)

Btw I am curious, is logging level option not available in the master image ?

[makemake-kbo]

That shouldn't be happening. Internally Blutgang only uses methods supported by all EL clients.
Try the 0.3.5-debug-verbose docker image.
It should have more logging outputs that should help us debug this.

Sure, Let me run using this docker image. Thank you for pushing the image :)

Btw I am curious, is logging level option not available in the master image ?

No, its a compile time feature flag.

[nidhi-singh02]

I ran with the docker image, seems the chain head in blutgang is getting updated, not sure why BlockByNumber method from geth module does not shows the updated block, it is stuck at Block number 1 always, whereas BlockNumber works fine.

[makemake-kbo]

I ran with the docker image, seems the chain head in blutgang is getting updated, not sure why BlockByNumber method from geth module does not shows the updated block, it is stuck at Block number 1 always, whereas BlockNumber works fine.

smells like a cache issue, try clearing it and see if that fixes it.

[nidhi-singh02]

Are you referring to do_clear for cleaning the cache ?

[makemake-kbo]

Are you referring to do_clear for cleaning the cache ?
@nidhi-singh02

yes, either that or deleting the database folder.

[nidhi-singh02]

I have set do_clear parameter to true in the toml file, still same issue with BlockByNumber using reth full node or nethermind full node. Only geth full node works.

[makemake-kbo]

please try the 0.4.0 branch, if possible with debug-verbose on, or the 0.3.5-debug-verbose docker image. its hard to debug this issue without this context.