Implement best practices for linkerd-multicluster app

[LolloneS]

Coming from here.

The goal of this issue is to have the App eventually adhere to standard cloud-native best-practices. The points below are to be considered non-exhaustive. Feel free to add some that make sense and remove some that don't. If this issue does not make sense (e.g. is a duplicate or covers a deprecated app) please close it.

Expected outcome:

• the app has properly set PDBs (as percentages)
• the app can scale horizontally based on Custom Metrics (if it makes sense)
• the app runs with more than 1 replica by default (if it needs HA)
• the app does not use emptyDir volumes if not strictly necessary

[mcharriere]

the app has properly set PDBs (as percentages)
the app can scale horizontally based on Custom Metrics (if it makes sense)
the app runs with more than 1 replica by default (if it needs HA)

No need, no service actually running

the app does not use emptyDir volumes if not strictly necessary

app needs emptyDir enabled cluster-wide. This is a problem as our default policy is blocking that. see https://github.com/giantswarm/giantswarm/issues/21039