global memory controller fail to kill delete stmt for a long time

[XuHuaiyu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

|2023-12-20 12:51:52 | [2023/12/20 04:51:52.327 +00:00] [WARN] [servermemorylimit.go:106] ["global memory controller failed to kill the top-consumer in 481s"] [connID=1480195538757528957] ["sql digest"=20c3bbf1d0fb980ae06a8ff432bb7ab93712612b3174973b5e07e21855fd41bb] ["sql text"="delete   from xx.bu where `time`>='2023-12-19 12:00:00' and `time`<='2023-12-19 20:00:00'"] ["sql memory usage"=2550128640]
...
|2023-12-20 12:43:55 | [2023/12/20 04:43:55.926 +00:00] [WARN] [servermemorylimit.go:106] ["global memory controller failed to kill the top-consumer in 5s"] [connID=1480195538757528957] ["sql digest"=20c3bbf1d0fb980ae06a8ff432bb7ab93712612b3174973b5e07e21855fd41bb] ["sql text"="delete   from xx.bu where `time`>='2023-12-19 12:00:00' and `time`<='2023-12-19 20:00:00'"] ["sql memory usage"=2550128640]
|2023-12-20 12:43:50 | [2023/12/20 04:43:50.926 +00:00] [WARN] [servermemorylimit.go:148] ["global memory controller tries to kill the top1 memory consumer"] [connID=1480195538757528957] ["sql digest"=20c3bbf1d0fb980ae06a8ff432bb7ab93712612b3174973b5e07e21855fd41bb] ["sql text"="delete   from xx.bu where `time`>='2023-12-19 12:00:00' and `time`<='2023-12-19 20:00:00'"] [tidb_server_memory_limit=11510510320] ["heap inuse"=12261236736] ["sql memory usage"=2550128640]

heap profile:

goroutines:
goroutine2023-12-20T04_50_54Z.txt

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

v6.5.3

[XuHuaiyu]

By the way, the memory usage of kv.(*LockCtx).InitCheckExistencemight not be tracked by memory tracker

[ekexium]

4 problems here:

• 1. Some memory usage in the locking phase is not tracked by the tracker. We can use another issue to track the feautre/improvement Track memory usage in pessimistic locking phase #51217
• 2. Kill signal handling in the locking phase. This is the case in the flamegraph. In master the problem should have been fixed as the kill signal and cancel signal is unified.
• 3. Kill signal is not handled in commit phase.
  • feat: check kill signal in 2pc committer tikv/client-go#1084
• 4. Kill signal is an enumeration type(uint32) signal, but is often checked as killed == 1, which can mistakenly ignore other kill signals. The mistake occurs in both client-go and TiDB.
  • fix: check kill signal against 0 tikv/client-go#1102
  • fix: check kill signal against 0 #50029

We can use the issue to track the problem 2 and 3.

[seiya-annie]

/found customer