Enhanced forced deletion on karmadactl unjoin

[zhzhuang-zju]

What type of PR is this?
/kind feature

What this PR does / why we need it:
When karmadactl unjoin a member cluster, the unjoin may fail all the time because the resource deletion of the member cluster fails. By enhancing the forced deletion on karmadactl unjoin, if the resource deletion of the member cluster fails also force unjoin
Which issue(s) this PR fixes:
Fixes #4431

Special notes for your reviewer:

1. delete clusterRole of member1

➜  karmada git:(master) km1 delete clusterrole karmada-controller-manager:karmada-member1 
clusterrole.rbac.authorization.k8s.io "karmada-controller-manager:karmada-member1" deleted

2. karmadactl unjoin member1

➜  karmada git:(master) karmadactl unjoin member1 --cluster-kubeconfig=/root/.kube/members.config
I1219 15:56:28.602959   54698 unjoin.go:264] Waiting for the cluster object member1 to be deleted
...
I1219 15:57:27.605630   54698 unjoin.go:264] Waiting for the cluster object member1 to be deleted
E1219 15:57:27.605651   54698 unjoin.go:268] Failed to delete cluster object. cluster name: member1, error: timed out waiting for the condition
E1219 15:57:27.605675   54698 unjoin.go:191] Failed to delete cluster object. cluster name: member1, error: timed out waiting for the condition
error: timed out waiting for the condition

3. karmadactl unjoin member1 --force

➜  karmada git:(master) karmadactl unjoin member1 --cluster-kubeconfig=/root/.kube/members.config --force
I1219 15:57:48.816159   54808 unjoin.go:264] Waiting for the cluster object member1 to be deleted
...
I1219 15:58:47.822262   54808 unjoin.go:264] Waiting for the cluster object member1 to be deleted
E1219 15:58:47.822290   54808 unjoin.go:268] Failed to delete cluster object. cluster name: member1, error: timed out waiting for the condition
E1219 15:58:47.822305   54808 unjoin.go:191] Failed to delete cluster object. cluster name: member1, error: timed out waiting for the condition
I1219 15:58:47.822310   54808 unjoin.go:193] Start forced deletion by remove work finalizer. cluster name: member1
I1219 15:58:47.847819   54808 unjoin.go:200] Succeeded to remove work's finalizer. After confirming the success of the unjoin, manually delete remaining resources on the cluster member1.
➜  karmada git:(master) ka get cluster
NAME      VERSION   MODE   READY   AGE
member2   v1.27.3   Push   True    4h3m
member3   v1.27.3   Pull   True    4h3m

4. rejoin

➜  karmada git:(master) karmadactl join member1 --cluster-kubeconfig=/root/.kube/members.config             
cluster(member1) is joined successfully
➜  karmada git:(master) ka get cluster
NAME      VERSION   MODE   READY   AGE
member1   v1.27.3   Push   True    5s
member2   v1.27.3   Push   True    4h15m
member3   v1.27.3   Pull   True    4h15m

Does this PR introduce a user-facing change?:

`karmadactl unjoin`: force unjoin member clusters with the flag `--force`

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 50 lines in your changes missing coverage. Please review.

Project coverage is 38.17%. Comparing base (e117edc) to head (726b067).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/karmadactl/unjoin/unjoin.go	0.00%	50 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4451      +/-   ##
==========================================
+ Coverage   37.99%   38.17%   +0.18%     
==========================================
  Files         649      649              
  Lines       45133    45179      +46     
==========================================
+ Hits        17148    17248     +100     
+ Misses      26688    26623      -65     
- Partials     1297     1308      +11     

Flag	Coverage Δ
unittests	38.17% <0.00%> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chaunceyjiang]

This request I think is reasonable, but I don't think just deleting the cluster can solve it, it should also forcibly delete the work under this cluster. It should also forcibly delete other resources like token,secret,etc.

[zhzhuang-zju]

This request I think is reasonable, but I don't think just deleting the cluster can solve it, it should also forcibly delete the work under this cluster. It should also forcibly delete other resources like token,secret,etc.

In fact, I achieve the purpose of forced deletion by deleting the finalizer that blocks the deletion step. If the cluster object can be deleted successfully, then resources such as secrets and tokens can also be deleted successfully.

[zhzhuang-zju]

The issue can be resolved by setting PropagationPolicy.Spec.PreserveResourcesOnDeletion.

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lonelycz for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/karmadactl/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zhzhuang-zju]

The issue can be resolved by setting PropagationPolicy.Spec.PreserveResourcesOnDeletion.

Apologies for the confusion between the two scenarios earlier. PropagationPolicy.Spec.PreserveResourcesOnDeletion is used to retain resources in the member cluster when a workload is deleted. Meanwhile, karmadactl join --force is used to delete cluster and secret resources even if resources in the cluster targeted for unjoin are not removed successfully. I will track the progress of this feature through #5477.