Implement best practices for aws-cloud-controller-manager-app app

[LolloneS]

Coming from here.

The goal of this issue is to have the App eventually adhere to standard cloud-native best-practices. The points below are to be considered non-exhaustive. Feel free to add some that make sense and remove some that don't. If this issue does not make sense (e.g. is a duplicate or covers a deprecated app) please close it.

Expected outcome:

• the app has properly set PDBs (as percentages)
• the app can scale horizontally based on Custom Metrics (if it makes sense)
• the app runs with more than 1 replica by default (if it needs HA)
• the app does not use emptyDir volumes if not strictly necessary

[JosephSalisbury]

yo @T-Kukawka any chance we can close / kill here? the overarching issue is hanging

[T-Kukawka]

This is the daemonset with VPA enabled by default, each Control Plane is running separate instance, no volumes in use hence no EmptyDirs. No actions here, closing.