Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OP: fix flaky data race in e2e tests #528

Open
bap2pecs opened this issue Jul 21, 2024 · 0 comments
Open

OP: fix flaky data race in e2e tests #528

bap2pecs opened this issue Jul 21, 2024 · 0 comments

Comments

@bap2pecs
Copy link
Collaborator

bap2pecs commented Jul 21, 2024

I found a flaky test in #526

https://app.circleci.com/pipelines/github/babylonchain/finality-provider/1996/workflows/f0949f6d-2f37-4cbd-be03-bb0773cea599/jobs/5831 failed due to a flaky data race

data race
=== RUN   TestFinalityStuckAndRecover
    babylon_node_handler.go:153: babylon log file: /tmp/zBabylonTest2527062729/babylon.log
Starting babylond with command:  /home/circleci/.go_workspace/bin/babylond start --home=/tmp/zBabylonTest2527062729/node0/babylond --log_level=trace --trace
2024-07-20T23:38:00.999Z	INFO	cosmos/log.go:131	Successful transaction	{"chain_id": "chain-test", "gas_used": 2690989, "fees": "8016ubbn", "fee_payer": "\ufffd\ufffd\ufffdЬ\ufffd@\u0010\ufffd\"\u0005\ufffd\u000bTN$\ufffd\ufffd-\ufffd", "height": 2, "msg_types": ["/cosmwasm.wasm.v1.MsgStoreCode"], "tx_hash": "153B06308B4F1E2A64B4193BCA2C8FE5138C162750C9A9544D236577C1E2F895"}
2024-07-20T23:38:05.911Z	INFO	cosmos/log.go:131	Successful transaction	{"chain_id": "chain-test", "gas_used": 170224, "fees": "454ubbn", "fee_payer": "\ufffd\ufffd\ufffdЬ\ufffd@\u0010\ufffd\"\u0005\ufffd\u000bTN$\ufffd\ufffd-\ufffd", "height": 3, "msg_types": ["/cosmwasm.wasm.v1.MsgInstantiateContract"], "tx_hash": "05CF67BC2991E7A36028F5906B0AD94CCA3FE6FB969C6E593A9F541DD513391F"}
    op_test_manager.go:372: [23:38:05.913] op-finality-gadget contract address: bbn14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9sw76fy2
    setup.go:490: Generating L2 genesis l2_allocs_mode delta
==================
WARNING: DATA RACE
Write at 0x00c0080da060 by goroutine 1152692:
  github.com/ethereum-optimism/optimism/op-node/node.(*OpNode).initL2()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-node/node/node.go:404 +0x1529
  github.com/ethereum-optimism/optimism/op-node/node.(*OpNode).init()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-node/node/node.go:130 +0x3cd
  github.com/ethereum-optimism/optimism/op-node/node.New()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-node/node/node.go:107 +0x3c4
  github.com/ethereum-optimism/optimism/op-e2e.SystemConfig.Start()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-e2e/setup.go:737 +0x3dc8
  github.com/babylonchain/finality-provider/itest/opstackl2.startExtSystemsAndCreateConsumerCfg()
      /home/circleci/project/itest/opstackl2/op_test_manager.go:618 +0x7cd
  github.com/babylonchain/finality-provider/itest/opstackl2.StartOpL2ConsumerManager()
      /home/circleci/project/itest/opstackl2/op_test_manager.go:81 +0x129
  github.com/babylonchain/finality-provider/itest/opstackl2.TestFinalityStuckAndRecover()
      /home/circleci/project/itest/opstackl2/op_e2e_test.go:123 +0x44
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /usr/local/go/src/testing/testing.go:1648 +0x44

Previous read at 0x00c0080da060 by goroutine 1153399:
  github.com/ethereum-optimism/optimism/op-node/node.(*OpNode).OnNewL1Head()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-node/node/node.go:526 +0xf3
  github.com/ethereum-optimism/optimism/op-node/node.(*OpNode).OnNewL1Head-fm()
      <autogenerated>:1 +0x75
  github.com/ethereum-optimism/optimism/op-service/eth.WatchHeadChanges.func1()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-service/eth/heads.go:46 +0x361
  github.com/ethereum/go-ethereum/event.NewSubscription.func1()
      /home/circleci/.go_workspace/pkg/mod/github.com/ethereum-optimism/[email protected]/event/subscription.go:53 +0xd4

Goroutine 1152692 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1648 +0x82a
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1595 +0x238
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2052 +0x896
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1925 +0xb57
  main.main()
      _testmain.go:51 +0x2bd

Goroutine 1153399 (running) created at:
  github.com/ethereum/go-ethereum/event.NewSubscription()
      /home/circleci/.go_workspace/pkg/mod/github.com/ethereum-optimism/[email protected]/event/subscription.go:51 +0x1bc
  github.com/ethereum-optimism/optimism/op-service/eth.WatchHeadChanges()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-service/eth/heads.go:28 +0x177
  github.com/ethereum-optimism/optimism/op-node/node.(*OpNode).initL1.func1()
      /home/circleci/.go_workspace/pkg/mod/github.com/babylonchain/[email protected]/op-node/node/node.go:191 +0x1f4
  github.com/ethereum/go-ethereum/event.(*resubscribeSub).subscribe.func1()
      /home/circleci/.go_workspace/pkg/mod/github.com/ethereum-optimism/[email protected]/event/subscription.go:175 +0xb3
==================
    op_test_manager.go:200: [23:38:11.107] Fp home dir: /tmp/fpe2etest354798435/fp-home-0
    op_test_manager.go:258: Error checking balance: no balances found for address bbn1s6nf278lsn4eaevjzmgxmxhcm7sjv9mk5h7ada
    op_test_manager.go:258: Error checking balance: no balances found for address bbn1s6nf278lsn4eaevjzmgxmxhcm7sjv9mk5h7ada
    op_test_manager.go:264: [23:38:17.605] Sent 1000000ubbn to bbn1s6nf278lsn4eaevjzmgxmxhcm7sjv9mk5h7ada
    op_test_manager.go:200: [23:38:17.605] Fp home dir: /tmp/fpe2etest354798435/fp-home-1
    op_test_manager.go:258: Error checking balance: no balances found for address bbn1rcc8peydcuedta6xkhsam9vltryss600l5qzml
    op_test_manager.go:264: [23:38:22.156] Sent 1000000ubbn to bbn1rcc8peydcuedta6xkhsam9vltryss600l5qzml

more failures instances:

another info is this never reproduces locally. only fails on CI sometimes


maybe we can try adding a sleep here:

image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant