Websocket fork handling test instability

[stephenpdeos]

Integration test instability

https://github.com/eosnetworkfoundation/eos-evm-node/actions/runs/10407419480/job/28823465477

[taokayan]

Updates:

This issue mainly happened during shutdown process. I have once reproduced it in my laptop, and it seems to be the issue with the pipe:

2024-09-17T08:02:50.644445  Checking 0xb106d2c286183ffc3d1f0c4a6f0753bb20b407c2 balance

2024-09-17T08:02:50.645876  Checking 0x9e126c57330fa71556628e0aabd6b6b6783d99fb balance

2024-09-17T08:02:50.647202 checking /home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/eos_evm/eos-evm-node.stderr for errors

2024-09-17T08:02:50.648072 checking /home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/eos_evm/eos-evm-rpc.stderr for errors

2024-09-17T08:02:50.648183 test success, ready to shut down cluster

2024-09-17T08:02:50.648224 shutting down cluster

2024-09-17T08:02:50.648272  Test succeeded.

2024-09-17T08:02:50.648330   Cluster shutting down.

2024-09-17T08:02:50.648412    Killing node: ['/home/kayan-u20/workspaces/spring/build/bin/nodeos', '--blocks-dir', 'blocks', '--p2p-listen-endpoint', '0.0.0.0:9876', '--p2p-server-address', 'localhost:9876', '--p2p-peer-address', 'localhost:9776', '--p2p-peer-address', 'localhost:9877', '--p2p-peer-address', 'localhost:9878', '--plugin', 'eosio::producer_plugin', '--signature-provider', 'EOS7Uhj8J3PaiGhVscaxdESwSqidrcQ12ANpsiaLruwQNcnyheGbb=KEY:5JVEyt6QW32oa4bLyfhKTm99zFcGXr2A5QqCvD7gpwzp5rKj8Cd', '--signature-provider', 'PUB_BLS_QPi_ttnQFlVCGiCMNwdLf35YWqF9bGtXB7cI-poRkJUwKYJ7o4-NgMxeKag1kN0I-CLG_8-YdHE3QsPw_w0aMu9WYrkx1-xD1Le1bDDCC_uDH8lhsXP_Pz3U0ly7XUYUH5xkqA=KEY:PVT_BLS_lsUNCVOrXZzVxQ5FTvDFjz6KWEcSC2ZR9yMQSmGVDQQYhnbs', '--producer-name', 'defproducera', '--producer-name', 'defproducerb', '--producer-name', 'defproducerc', '--producer-name', 'defproducerd', '--producer-name', 'defproducere', '--producer-name', 'defproducerf', '--producer-name', 'defproducerg', '--producer-name', 'defproducerh', '--producer-name', 'defproduceri', '--producer-name', 'defproducerj', '--producer-name', 'defproducerk', '--producer-name', 'defproducerl', '--producer-name', 'defproducerm', '--producer-name', 'defproducern', '--producer-name', 'defproducero', '--producer-name', 'defproducerp', '--producer-name', 'defproducerq', '--producer-name', 'defproducerr', '--producer-name', 'defproducers', '--producer-name', 'defproducert', '--producer-name', 'defproduceru', '--plugin', 'eosio::net_plugin', '--plugin', 'eosio::chain_api_plugin', '--vote-threads', '4', '--max-transaction-time', '-1', '--abi-serializer-max-time-ms', '990000', '--p2p-max-nodes-per-host', '3', '--max-clients', '25', '--connection-cleanup-period', '15', '--plugin', 'eosio::producer_api_plugin', '--plugin', 'eosio::trace_api_plugin', '--trace-no-abis', '--contracts-console', '--resource-monitor-not-shutdown-on-threshold-exceeded', '--http-max-response-time-ms', '990000', '--config-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_00', '--data-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_00', '--genesis-json', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_00/genesis.json', '--genesis-timestamp', '2024-09-17T07:56:44.004', '--http-validate-host', 'false', '--http-server-address', 'localhost:8888']

2024-09-17T08:02:50.669025    Killing node: ['/home/kayan-u20/workspaces/spring/build/bin/nodeos', '--blocks-dir', 'blocks', '--p2p-listen-endpoint', '0.0.0.0:9877', '--p2p-server-address', 'localhost:9877', '--p2p-peer-address', 'localhost:9776', '--p2p-peer-address', 'localhost:9878', '--p2p-peer-address', 'localhost:9876', '--transaction-retry-max-storage-size-gb', '100', '--plugin', 'eosio::net_plugin', '--plugin', 'eosio::chain_api_plugin', '--vote-threads', '4', '--max-transaction-time', '-1', '--abi-serializer-max-time-ms', '990000', '--p2p-max-nodes-per-host', '3', '--max-clients', '25', '--connection-cleanup-period', '15', '--plugin', 'eosio::producer_api_plugin', '--plugin', 'eosio::trace_api_plugin', '--trace-no-abis', '--contracts-console', '--resource-monitor-not-shutdown-on-threshold-exceeded', '--http-max-response-time-ms', '990000', '--config-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_01', '--data-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_01', '--genesis-json', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_01/genesis.json', '--genesis-timestamp', '2024-09-17T07:56:44.004', '--http-validate-host', 'false', '--http-server-address', 'localhost:8889']

2024-09-17T08:02:50.679384    Killing node: ['/home/kayan-u20/workspaces/spring/build/bin/nodeos', '--blocks-dir', 'blocks', '--p2p-listen-endpoint', '0.0.0.0:9878', '--p2p-server-address', 'localhost:9878', '--p2p-peer-address', 'localhost:9776', '--p2p-peer-address', 'localhost:9876', '--transaction-retry-max-storage-size-gb', '100', '--plugin', 'eosio::net_plugin', '--plugin', 'eosio::chain_api_plugin', '--vote-threads', '4', '--max-transaction-time', '-1', '--abi-serializer-max-time-ms', '990000', '--p2p-max-nodes-per-host', '3', '--max-clients', '25', '--connection-cleanup-period', '15', '--plugin', 'eosio::producer_api_plugin', '--plugin', 'eosio::trace_api_plugin', '--trace-no-abis', '--contracts-console', '--resource-monitor-not-shutdown-on-threshold-exceeded', '--http-max-response-time-ms', '990000', '--plugin', 'eosio::state_history_plugin', '--state-history-endpoint', '127.0.0.1:8999', '--trace-history', '--chain-state-history', '--disable-replay-opts', '--config-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_02', '--data-dir', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_02', '--genesis-json', '/home/kayan-u20/workspaces/eos-evm-node/build/tests/TestLogs/nodeos_eos_evm_test12553/node_02/genesis.json', '--genesis-timestamp', '2024-09-17T07:56:44.004', '--http-validate-host', 'false', '--http-server-address', 'localhost:8890']

^CTraceback (most recent call last):

  File "/home/kayan-u20/workspaces/eos-evm-node/build/tests/./nodeos_eos_evm_test.py", line 1143, in <module>

    TestHelper.shutdown(cluster, walletMgr, testSuccessful=testSuccessful, dumpErrorDetails=dumpErrorDetails)

  File "/home/kayan-u20/workspaces/spring/build/tests/TestHarness/TestHelper.py", line 191, in shutdown

    cluster.shutdown()

  File "/home/kayan-u20/workspaces/spring/build/tests/TestHarness/Cluster.py", line 1421, in shutdown

    node.kill(signal.SIGTERM)

  File "/home/kayan-u20/workspaces/spring/build/tests/TestHarness/Node.py", line 290, in kill

    self.popenProc.wait()

  File "/usr/lib/python3.10/subprocess.py", line 1209, in wait

    return self._wait(timeout=timeout)

  File "/usr/lib/python3.10/subprocess.py", line 1959, in _wait

    (pid, sts) = self._try_wait(0)

  File "/usr/lib/python3.10/subprocess.py", line 1917, in _try_wait

    (pid, sts) = os.waitpid(self.pid, wait_flags)

KeyboardInterrupt

2024-09-17T08:11:19.257719 Shutting down wallet manager process 12674

[arhag]

The problem was discovered to be a bug in Spring (AntelopeIO/spring#815) and once the Spring bug is fixed, this issue should also be resolved.

[arhag]

Resolved by the Spring bug fix AntelopeIO/spring#816.