Add HA mode for service-mirror

[mateiidavid]

In certain scenarios, the service-mirror may act as a single point of failure. Linkerd's multicluster extension supports an --ha mode to increase reliability by adding more replicas, however, it is currently supported only in the gateway.

To avoid the service-mirror as a single point of failure, this change introduces an --ha flag for linkerd multicluster link. The HA flag will use a set of value overrides that will:

• Configure the service-mirror with affinity and PDB policies to ensure replicas are spread across hosts to protect against (in)voluntary disruptions;
• Configure the service-mirror to run with more than 3 replicas;
• Configure the service-mirror deployment's rolling strategy to ensure at least one replica is available.

Additionally, with the introduction of leader election, linkerd mc gateways displays redundant information since metrics are collected from each pod. This change adds a small lookup table of currently lease claimants. Metrics are extracted only for claimants.

Depends on #11046

[mateiidavid]

DIff:

---

# 
# < -- missing from unha.yaml
# > -- present in unha.yaml
:; diff ha.yaml unha.yaml
121c121
<   replicas: 3
---
>   replicas: 1
126,128d125
<   strategy:
<     rollingUpdate:
<       maxUnavailable: 1
167,182d163
< ---
< kind: PodDisruptionBudget
< apiVersion: policy/v1
< metadata:
<   name: linkerd-service-mirror-target
<   namespace: linkerd-multicluster
<   labels:
<     component: linkerd-service-mirror
<   annotations:
<     linkerd.io/created-by: linkerd/cli dev-352e404a-matei
< spec:
<   maxUnavailable: 1
<   selector:
<     matchLabels:
<       component: linkerd-service-mirror
<       mirror.linkerd.io/cluster-name: target

# Before
:; linkerd mc gateways
CLUSTER  ALIVE    NUM_SVC      LATENCY
target   False          2            -
target   True           2          3ms
target   False          2            -

# After
:; bin/linkerd mc gateways
CLUSTER  ALIVE    NUM_SVC      LATENCY
target   True           2          3ms

[risingspiral]

For a non HA mode with only one instance running will it still have a lease resource?

[mateiidavid]

Yup. This check should be safe to run regardless of the deployment model. The same applies for multicluster gateways where we pull metrics from the "leader". We can guarantee there will always be a leader since a lease is created and claimed under all circumstances.

As an aside, there might be a smarter way to pull metrics (e.g. aggregate metrics from all pods under a deployment). I thought I'd keep it simple though.

[alpeb]

Nice!
Can you also also include a podAntiAffinity field like we do for the linkerd-control-plane components? I believe the PDB doesn't cover that aspect.

[mateiidavid]

Added missing pod anti affinity partial in the deployment. For context:

• Anti affinity works by selecting over a group of pods (using a label selector) and a topology key (e.g. kubernetes.io/hostname). Depending on the strategy (preferred vs required) pods will not be scheduled on the same node that satisfies the topology key. This allows us to have separate failure domains for HA.
• Typically, a component or app label key is used to differentiate between pods (i.e. as the affinity label selector). All service-mirror components, irrespective of the cluster they're linked against, have the same value for "component"; component=linkerd-service-mirror
• Users should be able to selectively turn HA on on a link-by-link basis. Enforcing affinity using component means all service-mirror pods will be affected by the scheduling policy, not just service-mirror pods specific to that link.

As a result, I have chosen to use a different label selector. Full list of labels each service-mirror receives is below:

{
"component":"linkerd-service-mirror",
"linkerd.io/control-plane-ns":"linkerd",
"linkerd.io/extension":"multicluster",
"linkerd.io/proxy-deployment":"linkerd-service-mirror-target",
"linkerd.io/workload-ns":"linkerd-multicluster",
"mirror.linkerd.io/cluster-name":"target",
"pod-template-hash":"c7c6555df"
}

The only variable value is the link cluster name. As a result, anti affinity selects pods using mirror.linkerd.io/cluster-name=<name> as a label selector. Another alternative would be to use both mirror.linkerd.io/cluster-name and component, or better yet use component and append the cluster name in the label value.

[alpeb]

Thanks @mateiidavid, I think the pod anti-affinity scheme if fine enough; people could kustomize if they'd require something more sophisticated 👍