EBS BR support to multiple k8s cluster TiDB

[BornChanger]

Feature Request

Is your feature request related to a problem? Please describe:

TiDB operator already supports deployment in multiple k8s clusters, but AWS EBS snapshot based Backup/Restore doesn't support the deployment yet.

Describe the feature you'd like:

AWS EBS snapshot based BR needs to support TiDB cluster deployed in multiple k8s clusters. We would like to use the kubenetes federation solution, where federation operators in control plane cooperate tidb-operators in the data plane by Kubernetes API server. See #5004 for details

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

• Federation operators
  • setup the br-federation-manager (fed: setup the br-federation-manager #4996)
  • federation backup operator (br: implement federation volume backup controller #5013)
  • federation restore operator (br: implement federation volume restore controller  #5039)
  • federation backup schedule operator (Fed backup schedule #5036)
  • warm-up restored EBS volumes (br: add warmup process to volume snapshot restore #5229)
  • change to use one kubeconfig file (br: modify to use one kubeconfig for multiple data planes #5186)
  • shrink window of GC and schedling suspension (ebs br: resume gc and scheduler when volume snapshots created #5288)
  • add additional volumes to backup/restore/warmup pods (br: add additional volumes to backup/restore/warmup pods #5414)
  • serial execution of volume backup gc (ebs br: serial execution of volume backup gc #5452)
  • expose the volumebackup init pod ttl as a crd attribute (ebs br: expose the volumebackup init pod ttl as a crd attribute #5490)
  • Add annotations for warmup pod (br: Add annotations for warmup pod #5445)
  • backoff support for snapshots deletion (ebs br: backoff support for snapshots deletion #5492)
  • Add sleep before shutdown backup container (Add sleep before shutdown backup container #5454)
  • Add metrics for volume backup (br: add metrics for volume backup (#5534) #5545)
  • Add GC immune tag (ebs br: add gc immune flag (#5567) #5571)
  • Support multiple backup schedules (backup: support multiple schedules #5633)
• Data plane operator refactoring
  • data plane backup process refactoring (br: refactor volume snapshot backup to support across k8s clusters #4999)
  • data plane restore process refactoring (br: refactor restore controller to support across k8s #5010)
  • Add operator tags for restored volume (pkg: add 2 operator related tags to restored ebs volume #5104)
  • Take cluster manifest backup along with data (br: backup manifests related to tidb cluster in EBS backup #5207)
  • Add incremental volume snapshot backup size(br: add incremental backup size for volume snapshot backup #5188)
  • Make snapshot size calculation optional (ebs br: make snapshot size calculation optional #5202)
  • Warmup checkpointing support (backup: added ebs-warmup for restoring from EBS snapshots #5238)
  • Remove k8s cluster name from restore/warmup pod names (Remove k8s cluster name from restore/warmup pod names #5318)
  • add create pv permissions to operator (ebs br: add create pv permissions to operator #5314)
  • Warmup strategy frame support (added warmup strategy #5315)
  • FSR warmup operator support (Fsr warmup support #5338)
  • FSR warmup api quota control (ebs br: control the snapshots batch size for fsr enable/disable tidb#48506)
  • FSR credit balance check (ebs br: make sure fsr credit is full filled tidb#48627)
  • Remove k8s cluster name from backup/init pod names([BR] Remove k8s cluster name from backup/init pod names #5418)
  • Support br-federation-manager managing resources in specific-namespace tidb clusters(Support br-federation-manager managing resources in specific-namespace tidb clusters #5410)
  • Support to terminate sidecars in BR job (Add trap to create a path when BR container exit #5431)
  • add resilience to single TiKV crash (ebs br: add resilience to single TiKV crash (#5572) #5585)
  • test WAL files while warming up (warmup-image: test WAL files while warming up (#5566) #5593)
  • Skip corruption check on last WAL (Skip corruption check on last WAL (#5596) #5605)
  • Add sleep before shutdown backup container (Add sleep before shutdown backup container (#5454) #5606)
  • add pv name in restore EBS vol ([BR] add pv name in restore EBS vol (#5612) #5615)
  • Make warmup failure/skip on corruption configurable (Make warmup failure/skip on corruption configurable (#5622) #5635)
  • Fail restore if warmup fails only during check-wal-only strategy (Fail restore if warmup fails only during check-wal-only strategy (#5621) #5636)
• BR command line refactoring
  • tidb br command support to keep safepoint and pause PD scheduling (br/cmd: implement the operator pause-gc-and-schedulers command tidb#43562)
  • tidb br command support to do volume snapshot only (br/backup_ebs: added flag operator-paused-gc-and-scheduler. tidb#43687)
  • Use AWS API CreateSnapshots instead of CreateSnapshot to create snapshots for multiple volumes (br: use AWS CreateSnapshots API to automic create snapshots for multiple volumes tidb#43591)
  • Cross-AZ restore support (br : a new flag to enable ebs br cross az restore tidb#43962)
  • Tagging to snapshots and restored volumes (br: add tags to snapshots and restored volumes tidb#43933 and br: ebs tags refactoring tidb#44381)
  • Add retry for aws service throttling (ebs_backup: added retry for exceeding of quota tidb#44328)
  • Add retry in data recovery phase (snap_restore: added retry for recovery tidb#46094))
  • FSR warmup BR support (ebs br: provide fsr warmup to tikv data volumes tidb#47272)
  • suppport encryted ebs volumes in restore (BR: Support encryption for restored ebs volumes tidb#48900)
  • Allow porting labels to restore pod (Allow porting labels to restore pod #5349)
  • skip wait apply phase of restore (restore_data: remove wait apply phase tidb#50316)
  • allow temporary TiKV unreachable during starting snapshot backup (ebs_br: allow temporary TiKV unreachable during starting snapshot backup tidb#49154)
  • remove wait apply (restore_data: remove wait apply phase tidb#50316)
  • establish connection to all stores before pausing admin (prepare_snap: establish connection to all stores before pausing admin tidb#51449)
  • pause scheduler after all connections established (operator: pause scheduler after all connections established tidb#51823)
  • remove checking store number (disk_snap_restore: remove checking store number tidb#51886)
  • make an infinity retry for connecting to store (operator: make an infinity retry for connecting to store tidb#52177)
• Serviceability
  • Report full backup size for all backup jobs including failed job (ebs: calc full snapshot size and report backup size for failed backup #5007)
  • Report flashback kv read/write metrics (command: distinguish different states in the flashback tag statistics tikv/tikv#14792)
  • Record elapsed time for snapshot size calculation (ebs br: print out the elapsed time for each snapshot size calculation #5263)
  • Design doc in gitHub (docs: add design doc for ebs br support to multiple k8s clusters #5004)
  • release document update (en: add doc about volume snapshot backup/restore across multiple kubernetes docs-tidb-operator#2406)
  • 1.5 release document (en: add doc about volume snapshot backup/restore across multiple kubernetes (#2406) docs-tidb-operator#2446)
  • print goroutines at init pod exit (operator: print callstack when exiting before preparing done tidb#51371)
  • expose track-and-verify-wals-in-manifest config (rocksdb: Expose track-and-verify-wals-in-manifest config tikv/tikv#16546)
  • print callstack when exiting before preparing done (operator: print callstack when exiting before preparing done tidb#51371)
• Bug fixes** from 7/17 to nowPending issues ( area/ebs-br )
  • pass region name for aws EBS volume operation (ebs br: pass region name for aws EBS volume operation #5195)
  • when there are message dropping, snap_restore may fail (snap_recovery: added keep leader tikv/tikv#15124 snap-restore: added the config for temporarily disable check quorum tikv/tikv#15196)
  • snap_restore: cannot elect leader when start up (snap_restore: fix snapshot recovery leader election tikv/tikv#15297)
  • resend recover_region while there are TiKV restarts (snap_restore: resend recover_region while there are TiKV restarts tidb#45361)
  • EBS backup clean successfully when the backupmeta is lost (ebs br: fail delete cr operation if backup meta is missing #5199)
  • Delete a running VolumeBackup, backup CRs are deleted, but metadata and EBS snapshots are still remained (ebs br: fail delete cr operation if backup meta is missing #5199)
  • ebs br: backup timetaken is wrong (ebs br: backup timetaken is wrong #5212)
  • Need to add the restore name to the warmup job name, otherwise it will affect the next restore task (br: add warmup process to volume snapshot restore #5229)
  • Avoid to allocate multiple warmup pods to the same node (br: add warmup process to volume snapshot restore #5229)
  • Add retry logic for snapshot size calculation (ebs br: add retry support for ListSnapshotBlocks() and ListChangedBlocks() #5232)
  • Add label support for IAM ROLE (allow porting labels when initiating backup/restore member #5241)
  • Don't do infinite retry to pause pd scheduler at backup initial phase (br: avoid possible infinite loop checking tikv gc and scheduling status tidb#46078)
  • Killed warmup pod is not auto restarted (br: add warmup process to volume snapshot restore #5229)
  • No complete time field in restore cr (br: modify time complete of restore status #5248)
  • avoid to hold global mutex while syncing titan manifest (titan: update titan to avoid manifest mutex on release-6.5 tikv/tikv#15399)
  • add executable mode to file scanner warmup (ebs-warmup: added exec mode for warmup-steps.sh #5258)
  • TiKV start fails with "delete blob file twice" error (Update titan to fix possible delete blob file twice error tikv/tikv#15470)
  • Performance jitter on restored 100T cluster (ebs br: for a 100T cluster after restore, the QPS decreased and became unstable in the first two hours #5269)
  • file level warmup properly exited once SIGINT received (ebs-warmup: added test cases && better exit logic #5272)
  • data plane backup job could suspend due to stderr pipeline is filled (ebs br: resume gc and scheduler when volume snapshots created #5288)
  • Check the tikv volume # mismatch (pkg: validate volume number for ebs snapshot restore #5292)
  • Restore panic when concurrent running with lightning local backend mode (snapshot_backup: deny executing tidb-lightning import while running snapshot_backup tidb#47001, sst_importer: impl SuspendImport interface tikv/tikv#15612)
  • delete restore data pod when restoring will cause restore progress stuck (snap_restore: Abort last recover region tikv/tikv#15685)
  • allow porting labels when initiating backup/restore member (allow porting labels when initiating backup/restore member #5241)
  • tikv panic during restoring ( snapshot_backup: enhanced prepare stage tikv/tikv#15946 backup: advacned prepare implementation tidb#48439)
  • VolumeBackup failed due to resuming GC and scheduling (ebs br: mark backup/restore failed when job failed #5332)
  • Extend snapshot size calculation backoff (ebs br: extend throttling backoff #5370)
  • Backup failed at GC keeper due to huge resolved_ts gap (raftstore: gc uninitialized stale peer after merge (#15934) tikv/tikv#15937
    raftstore: check stale peer on leader missing (#16038) tikv/tikv#16044, raftstore: update apply state even if peer is removed (#16060) tikv/tikv#16081)
  • volume back schedule fail if the backup schedule has been activated after being paused for a long time(ebs br: volume back schedule fail if the backup schedule is activated after being paused for a long time #5392)
  • fix backup stuck due to init pod creating stuck (br: fix backup stuck due to init pod creating stuck #5457)
  • PV creation failed since PVC with the same name exists (br: fix pv not created due to same pvc ref name in different namespace #5417)
  • make sure backup can be resumed even after being paused for a long time (ebs br: make sure backup can be resumed even after being paused for a long time #5464)
  • fix file creating error due to directory not exists (br: fix file creating error due to directory not exists #5469)
  • fix redundant pvc and pv in metadata (br: fix redundant pvc and pv in metadata #5461)
  • fix other backup member init job check (br: fix other backup member init job check #5479)
  • resume gc and pd schedule asap if data plane backup fails (ebs br: resume gc and pd schedule asap if data plane backup fails #5491)
  • set volume backup failed if one data plane failed (br: set volume backup failed if one data plane failed #5500)
  • always save checkpoint to the warmup folder (always save checkpoint to the warmup folder #5507)
  • Fixing volume tagging (Fixing volume tagging #5519)
  • Use CreationTimestamp of volumebackup for gc check (Use CreationTimestamp of volumebackup for gc check #5518)
  • Tolerate to rolling restart (Added a new PrepareBackup RPC for snapshot backup kvproto#1204, snapshot_backup: enhanced prepare stage tikv/tikv#15946 and backup: advacned prepare implementation tidb#48439)
  • Fixing tagging to snapshots and restore volumes (ebs: adjust volume tagging #5525 and
    ebs br: new snapshot tagging tidb#50548)
  • Possible tag number exceeds aws quota (ebs br: handle tag limit exceed issue tidb#50941)
  • Mitigate bad RTO due to raft log gc at upstream cluster (disk_snapshot_backup: pause raft log GC while backing up tikv/tikv#16519)
  • Fail restore if any warmup job failed for volume-snapshot restores (Fail restore if any warmup job failed for volume-snapshot restores (#5569) #5578)
  • clean volumes when restore volume failed (br: clean volumes when restore volume failed (#5634) #5639)
  • initialize status metric to zero ([BR]initialize status metric to zero (#5637) #5648)
  • make gRPC connections synced (operator: make gRPC connections synced tidb#52051)
  • make an infinity retry for connecting to store (operator: make an infinity retry for connecting to store tidb#52177)
  • fix stuck when terminating (br/operator: fix stuck when terminating tidb#52264)
  • fix adapt env for snapshot backup stuck when encountered error (br/operator: fix adapt env for snapshot backup stuck when encountered error tidb#52607)

[BornChanger]

/assign @WangLe1321 @BornChanger @YuJuncen @csuzhangxc

[zhangjinpeng87]

Please don't include internal google doc link, please upload the design doc to github

[BornChanger]

Please don't include internal google doc link, please upload the design doc to github

Sure.