Add support to customize spare replicas during VolumeReplace

[anish-db]

What problem does this PR solve?

Currently the VolumeReplace feature assumes a default spare replica of 1 for PD & TiKV.
Sometimes it is useful to change this number. Setting a larger spare replica number helps for faster tikv region transfer
In multi-az setups, it is possible to get zone skewness with only a single replica, which will cause issues during scale down of spare replica.

Example multi-az senario:
3 zones , 6 replicas, initial zones per replica: 1,2,3,1,2,3
After adding 1 spare replica: 1,2,3,1,2,3,1
Now replace will start replacing from start, but now the new disks can get different zones, example:
after replacing first disk it could now become 2,2,3,1,2,3,1
After this if we scale down the spare replica (last one), it will become 2,2,3,1,2,3 which is zone-skewed and can be blocked depending on configured topology constraints.

In this case setting spareReplicas to 3 will avoid this issue

What is changed and how does it work?

Add a config option in TidbCluster spec, inside TiKVSpec and PDSpec to configure spare replicas

Code changes

• Has Go code change
• Has CI related scripts change

Tests

Re ran the manual test described in : #5150
Ran once as it is, and observed default spare replica of 1 was used.
Ran once by setting tikv spare replicas to 3, and observed 3 spare replicas was used

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

VolumeReplace feature now supports customizing the number of spare replicas used during TiKV or PD disk replacements inside the TidbCluster spec.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csuzhangxc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [csuzhangxc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-06-28 01:41:39.424623427 +0000 UTC m=+943025.910112244: ☑️ agreed by csuzhangxc.

[csuzhangxc]

/run-pull-e2e-kind

[csuzhangxc]

/cherry-pick release-1.6

[csuzhangxc]

/cherry-pick release-1.5

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.6: #5673.

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.5: #5674.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.