test: refactor real cluster test and implement cluster id case

[okJiang]

What problem does this PR solve?

Issue Number: Close #8683, close #8684

What is changed and how does it work?

• Using Golang to deploy cluster. In this way, we can deploy and destroy cluster depend on our test demand
• Using tag to manage different cluster between different test case. Different clusters do not affect each other
• Implement cluster id test case Implement cluster id test case in real cluster test #8684

Check List

Tests

• Integration test

make test-real-cluster
# testing with the real cluster...
cd /root/code/okjiang/pd/tests/integrations/realcluster && make check
make[1]: Entering directory `/root/code/okjiang/pd/tests/integrations/realcluster'
CGO_ENABLED=1 go test ./... -v -tags deadlock -race -cover || (\
echo "follow is pd-0 log\n" ; \
cat ~/.tiup/data/pd_real_cluster_test/pd-0/pd.log ; \
echo "follow is pd-1 log\n" ; \
cat ~/.tiup/data/pd_real_cluster_test/pd-1/pd.log ; \
echo "follow is pd-2 log\n" ; \
cat ~/.tiup/data/pd_real_cluster_test/pd-2/pd.log ; \
exit 1)
=== RUN   TestClusterID
=== RUN   TestClusterID/TestClientClusterID
[2024/10/10 16:47:18.115 +08:00] [INFO] [real_cluster.go:63] ["start to deploy a real cluster"]
[2024/10/10 16:47:18.116 +08:00] [INFO] [real_cluster.go:113] [/root/code/okjiang/pd/tests/integrations/realcluster]
[2024/10/10 16:47:28.124 +08:00] [INFO] [real_cluster.go:146] ["start to wait TiUP ready"] [tag=pd_real_cluster_test_cluster_id_0]
Error: Post "http://127.0.0.1:33032/command": dial tcp 127.0.0.1:33032: connect: connection refused
[2024/10/10 16:47:28.500 +08:00] [INFO] [real_cluster.go:154] ["TiUP is not ready, will retry"] ["retry times"=0] [tag=pd_real_cluster_test_cluster_id_0] [error="exit status 1"]
Pid   Role     Uptime
---   ----     ------
2946  pd       14.960070143s
2954  pd       14.912455918s
2961  pd       14.867212079s
2979  tikv     14.814994144s
2990  tikv     14.773555858s
3032  tikv     14.72004818s
3040  tidb     14.678945795s
3730  tiflash  3.642770313s
[2024/10/10 16:47:33.764 +08:00] [INFO] [real_cluster.go:150] ["TiUP is ready"] [tag=pd_real_cluster_test_cluster_id_0]
[2024/10/10 16:47:33.764 +08:00] [INFO] [real_cluster.go:63] ["start to deploy a real cluster"]
[2024/10/10 16:47:33.764 +08:00] [INFO] [real_cluster.go:113] [/root/code/okjiang/pd/tests/integrations/realcluster]
[2024/10/10 16:47:43.772 +08:00] [INFO] [real_cluster.go:146] ["start to wait TiUP ready"] [tag=pd_real_cluster_test_cluster_id_1]
Error: Post "http://127.0.0.1:36151/command": dial tcp 127.0.0.1:36151: connect: connection refused
[2024/10/10 16:47:44.082 +08:00] [INFO] [real_cluster.go:154] ["TiUP is not ready, will retry"] ["retry times"=0] [tag=pd_real_cluster_test_cluster_id_1] [error="exit status 1"]
Pid   Role     Uptime
---   ----     ------
5102  pd       14.919982227s
5111  pd       14.896268836s
5112  pd       14.876884134s
5122  tikv     14.860474382s
5128  tikv     14.84466765s
5151  tikv     14.82536701s
5175  tidb     14.808416266s
5843  tiflash  4.756919588s
[2024/10/10 16:47:49.369 +08:00] [INFO] [real_cluster.go:150] ["TiUP is ready"] [tag=pd_real_cluster_test_cluster_id_1]
[2024/10/10 16:47:50.409 +08:00] [INFO] [pd_service_discovery.go:638] ["[pd] close pd service discovery client"]
[2024/10/10 16:47:50.411 +08:00] [INFO] [real_cluster.go:72] ["start to destroy a real cluster"] [tag=pd_real_cluster_test_cluster_id_1]
[2024/10/10 16:47:50.448 +08:00] [INFO] [real_cluster.go:106] ["destroy success"] [pid="5063\n"]
[2024/10/10 16:47:55.452 +08:00] [INFO] [real_cluster.go:72] ["start to destroy a real cluster"] [tag=pd_real_cluster_test_cluster_id_1]
sh: line 0: kill: (7173) - No such process
[2024/10/10 16:47:55.498 +08:00] [INFO] [real_cluster.go:106] ["destroy success"] [pid="7173\n"]
[2024/10/10 16:48:00.503 +08:00] [INFO] [real_cluster.go:72] ["start to destroy a real cluster"] [tag=pd_real_cluster_test_cluster_id_0]
[2024/10/10 16:48:00.548 +08:00] [INFO] [real_cluster.go:106] ["destroy success"] [pid="2909\n"]
--- PASS: TestClusterID (47.47s)
    --- PASS: TestClusterID/TestClientClusterID (47.46s)
PASS
coverage: 56.2% of statements
ok      github.com/tikv/pd/tests/integrations/realcluster       48.686s coverage: 56.2% of statements

Release note

None.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign siddontang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[okJiang]

these cases will be updated with new style in the next pr

[okJiang]

what do you think? Do I need update them in this pr? @rleungx

[rleungx]

Both are ok to me.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.66%. Comparing base (48e72f1) to head (d1f9143).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8685      +/-   ##
==========================================
- Coverage   71.70%   71.66%   -0.04%     
==========================================
  Files         517      517              
  Lines       67450    67454       +4     
==========================================
- Hits        48364    48344      -20     
- Misses      15521    15539      +18     
- Partials     3565     3571       +6     

Flag	Coverage Δ
unittests	71.66% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[okJiang]

pull-integration-realcluster-test should be passed after PingCAP-QE/ci#3168 merge

[okJiang]

We will use third_bin after PingCAP-QE/ci#3168 merge