Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: data race in CI (in mysql_buffer.go) #18880

Closed
1 task done
ouyuanning opened this issue Sep 20, 2024 · 3 comments
Closed
1 task done

[Bug]: data race in CI (in mysql_buffer.go) #18880

ouyuanning opened this issue Sep 20, 2024 · 3 comments
Assignees
@ouyuanning
Labels
bug/ut kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone
2.0.0

Comments

@ouyuanning
Copy link
Contributor

ouyuanning commented Sep 20, 2024 •
edited
Loading

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

main

Commit ID

main

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

https://github.com/matrixorigin/matrixone/actions/runs/10952357425/job/30410915911?pr=18650

2024/09/20 03:02:06.132900 +0000 INFO logutil/action.go:35 start to close components {"service": "1-cn-3", "action": "close", "components": "cnservice/frontend"}
==================
WARNING: DATA RACE
Write at 0x00c0a275fe00 by goroutine 8066:
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close.func1.1()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:153 +0x172
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.23.0/x64/src/runtime/panic.go:605 +0x5d
  sync.(*Once).doSlow()
      /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:76 +0xe1
  sync.(*Once).Do()
      /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:67 +0x44
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:148 +0x8f
  github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).Stop()
      /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:117 +0x38b
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).serverShutdown()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:516 +0xb1
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).stopFrontend()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:345 +0xb2
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).Close()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:263 +0xfb
  github.com/matrixorigin/matrixone/pkg/embed.(*operator).Close()
      /home/runner/work/matrixone/matrixone/pkg/embed/operator.go:129 +0xf0
  github.com/matrixorigin/matrixone/pkg/embed.(*cluster).Close()
      /home/runner/work/matrixone/matrixone/pkg/embed/cluster.go:145 +0x166
  github.com/matrixorigin/matrixone/pkg/embed.TestMultiClusterCanWork()
      /home/runner/work/matrixone/matrixone/pkg/embed/cluster_test.go:61 +0xc2
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x44

Previous read at 0x00c0a275fe00 by goroutine 13452:
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).BeginPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:546 +0x51
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).beginPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2733 +0x56
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).appendPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2743 +0x40
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).SendColumnDefinitionPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2175 +0xec
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).WriteColumnDef()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:437 +0x58
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).respColumnDefsWithoutFlush()
      /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:239 +0x4e2
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).RespPreMeta()
      /home/runner/work/matrixone/matrixone/pkg/frontend/resp_client.go:120 +0xc5
  github.com/matrixorigin/matrixone/pkg/frontend.executeResultRowStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:51 +0x5cf
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2695 +0x1912
  github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2558 +0x4a8
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x3cc
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x966

Expected Behavior

No response

Steps to Reproduce

https://github.com/matrixorigin/matrixone/actions/runs/10952357425/job/30410915911?pr=18650

Additional information

No response
@ouyuanning ouyuanning added kind/bug Something isn't working needs-triage severity/s1 High impact: Logical errors or data errors that must occur labels Sep 20, 2024
@ouyuanning ouyuanning added this to the 1.3.0 milestone Sep 20, 2024
@ouyuanning
Copy link
Contributor Author

job-logs (7).txt.zip
log

@ouyuanning ouyuanning changed the title [Bug]: data race in CI [Bug]: data race in CI (in mysql_buffer.go) Sep 20, 2024
@sukki37 sukki37 added severity/s0 Extreme impact: Cause the application to break down and seriously affect the use bug/ut and removed severity/s1 High impact: Logical errors or data errors that must occur labels Sep 20, 2024
@sukki37
Copy link
Contributor

sukki37 commented Sep 20, 2024

repro：https://github.com/matrixorigin/matrixone/actions/runs/10953064595/job/30412774046?pr=18882

log：
job-logs.txt.zip

2024-09-20T04:29:29.7900908Z ==================
2024-09-20T04:29:29.7901355Z WARNING: DATA RACE
2024-09-20T04:29:29.7902002Z Write at 0x00c083fed110 by goroutine 2497:
2024-09-20T04:29:29.7903074Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close.func1.1()
2024-09-20T04:29:29.7904571Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:153 +0x172
2024-09-20T04:29:29.7905598Z   runtime.deferreturn()
2024-09-20T04:29:29.7906570Z       /opt/hostedtoolcache/go/1.23.0/x64/src/runtime/panic.go:605 +0x5d
2024-09-20T04:29:29.7907459Z   sync.(*Once).doSlow()
2024-09-20T04:29:29.7908320Z       /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:76 +0xe1
2024-09-20T04:29:29.7909318Z   sync.(*Once).Do()
2024-09-20T04:29:29.7910151Z       /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:67 +0x44
2024-09-20T04:29:29.7911238Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close()
2024-09-20T04:29:29.7912666Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:148 +0x8f
2024-09-20T04:29:29.7913962Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).Stop()
2024-09-20T04:29:29.7915367Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:117 +0x38b
2024-09-20T04:29:29.7916721Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).serverShutdown()
2024-09-20T04:29:29.7918239Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:516 +0xb1
2024-09-20T04:29:29.7919786Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).stopFrontend()
2024-09-20T04:29:29.7921253Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:345 +0xb2
2024-09-20T04:29:29.7922508Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).Close()
2024-09-20T04:29:29.7923928Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:263 +0xfb
2024-09-20T04:29:29.7925167Z   github.com/matrixorigin/matrixone/pkg/embed.(*operator).Close()
2024-09-20T04:29:29.7926535Z       /home/runner/work/matrixone/matrixone/pkg/embed/operator.go:129 +0xf0
2024-09-20T04:29:29.7927749Z   github.com/matrixorigin/matrixone/pkg/embed.(*cluster).Close()
2024-09-20T04:29:29.7929275Z       /home/runner/work/matrixone/matrixone/pkg/embed/cluster.go:145 +0x166
2024-09-20T04:29:29.7930569Z   github.com/matrixorigin/matrixone/pkg/embed.TestMultiClusterCanWork()
2024-09-20T04:29:29.7932062Z       /home/runner/work/matrixone/matrixone/pkg/embed/cluster_test.go:60 +0x93
2024-09-20T04:29:29.7933033Z   testing.tRunner()
2024-09-20T04:29:29.7934110Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
2024-09-20T04:29:29.7935031Z   testing.(*T).Run.gowrap1()
2024-09-20T04:29:29.7936199Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x44
2024-09-20T04:29:29.7936917Z 
2024-09-20T04:29:29.7937237Z Previous read at 0x00c083fed110 by goroutine 7924:
2024-09-20T04:29:29.7938312Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).BeginPacket()
2024-09-20T04:29:29.7940023Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:546 +0x51
2024-09-20T04:29:29.7941503Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).beginPacket()
2024-09-20T04:29:29.7943143Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2733 +0x56
2024-09-20T04:29:29.7944659Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).appendPacket()
2024-09-20T04:29:29.7946294Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2743 +0x40
2024-09-20T04:29:29.7947904Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).WriteEOFIFAndNoFlush()
2024-09-20T04:29:29.7950007Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:419 +0x6d
2024-09-20T04:29:29.7951576Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).respColumnDefsWithoutFlush()
2024-09-20T04:29:29.7953319Z       /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:254 +0x538
2024-09-20T04:29:29.7954880Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).RespPreMeta()
2024-09-20T04:29:29.7956531Z       /home/runner/work/matrixone/matrixone/pkg/frontend/resp_client.go:120 +0xc5
2024-09-20T04:29:29.7957922Z   github.com/matrixorigin/matrixone/pkg/frontend.executeResultRowStmt()
2024-09-20T04:29:29.7959669Z       /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:51 +0x5cf
2024-09-20T04:29:29.7960961Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmt()
2024-09-20T04:29:29.7962464Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2695 +0x1912
2024-09-20T04:29:29.7963820Z   github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt()
2024-09-20T04:29:29.7965326Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2558 +0x4a8
2024-09-20T04:29:29.7966791Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt()
2024-09-20T04:29:29.7968412Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x3cc
2024-09-20T04:29:29.7970118Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace()
2024-09-20T04:29:29.7971738Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x966
2024-09-20T04:29:29.7973160Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn()
2024-09-20T04:29:29.7974723Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2374 +0x213
2024-09-20T04:29:29.7976196Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse()
2024-09-20T04:29:29.7977839Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2338 +0x5d5
2024-09-20T04:29:29.7979315Z   github.com/matrixorigin/matrixone/pkg/frontend.doComQuery()
2024-09-20T04:29:29.7980846Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2945 +0x37ed
2024-09-20T04:29:29.7982189Z   github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest()
2024-09-20T04:29:29.7983674Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:3020 +0xd91
2024-09-20T04:29:29.7985134Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest()
2024-09-20T04:29:29.7986640Z       /home/runner/work/matrixone/matrixone/pkg/frontend/routine.go:290 +0xb39
2024-09-20T04:29:29.7987971Z   github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler()
2024-09-20T04:29:29.7989764Z       /home/runner/work/matrixone/matrixone/pkg/frontend/routine_manager.go:383 +0x55e
2024-09-20T04:29:29.7991389Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest()
2024-09-20T04:29:29.7993001Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:471 +0x284
2024-09-20T04:29:29.7994357Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage()
2024-09-20T04:29:29.7995799Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:440 +0x4b
2024-09-20T04:29:29.7997120Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop()
2024-09-20T04:29:29.7998567Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:203 +0x4b
2024-09-20T04:29:29.8000049Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn()
2024-09-20T04:29:29.8001562Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:199 +0x5dc
2024-09-20T04:29:29.8002978Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept.gowrap2()
2024-09-20T04:29:29.8004496Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:167 +0x4f
2024-09-20T04:29:29.8005239Z 
2024-09-20T04:29:29.8005476Z Goroutine 2497 (running) created at:
2024-09-20T04:29:29.8006121Z   testing.(*T).Run()
2024-09-20T04:29:29.8007098Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x825
2024-09-20T04:29:29.8007987Z   testing.runTests.func1()
2024-09-20T04:29:29.8009161Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2168 +0x85
2024-09-20T04:29:29.8010187Z   testing.tRunner()
2024-09-20T04:29:29.8011181Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
2024-09-20T04:29:29.8012055Z   testing.runTests()
2024-09-20T04:29:29.8013124Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2166 +0x8be
2024-09-20T04:29:29.8014007Z   testing.(*M).Run()
2024-09-20T04:29:29.8014981Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2034 +0xf17
2024-09-20T04:29:29.8015847Z   main.main()
2024-09-20T04:29:29.8016377Z       _testmain.go:73 +0x164
2024-09-20T04:29:29.8016738Z 
2024-09-20T04:29:29.8016968Z Goroutine 7924 (running) created at:
2024-09-20T04:29:29.8017987Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept()
2024-09-20T04:29:29.8019674Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:167 +0x1a8
2024-09-20T04:29:29.8021088Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startListener.gowrap1()
2024-09-20T04:29:29.8022652Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:139 +0x4f
2024-09-20T04:29:29.8023562Z ==================
2

@daviszhen daviszhen mentioned this issue Sep 21, 2024
Merged
7 tasks
@daviszhen
Copy link
Contributor

it was fixed

@daviszhen daviszhen assigned ouyuanning and unassigned daviszhen Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ouyuanning ouyuanning

Labels
bug/ut kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Milestone
2.0.0
Development

No branches or pull requests
4 participants
@ouyuanning @daviszhen @sukki37 @matrix-meow