Add periodic check for metric client health

[floreks]

Resolves #2306 and resolves #2251.

Added new flag metric-client-check-period that defines how often health check against configured metric client will be run. By default it is set to 30 seconds.

[codecov]

Codecov Report

Merging #2355 into master will decrease coverage by 0.06%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master   #2355      +/-   ##
=========================================
- Coverage   58.97%   58.9%   -0.07%     
=========================================
  Files         594     594              
  Lines       12757   12767      +10     
=========================================
- Hits         7523    7520       -3     
- Misses       5041    5055      +14     
+ Partials      193     192       -1

Impacted Files	Coverage Δ
.../backend/integration/metric/heapster/restclient.go	0% <ø> (ø)	⬆️
src/app/backend/resource/dataselect/dataselect.go	22.58% <ø> (ø)	⬆️
src/app/backend/integration/metric/manager.go	56.36% <0%> (-25.22%)	⬇️
src/app/backend/sync/secret.go	69.23% <0%> (-2.57%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b29ee1...db6fa79. Read the comment docs.

[cheld]

@floreks BTW: there was request from core to replace flags with config map

[floreks]

We can not replace them as Dashboard has to work also outside of the cluster. We could only add support for ConfigMap.

[cheld]

@floreks

The main complain about params is that:

• services must be restarted on change
• not versioned
• not managed by kubernetes

I guess it could be as a configuration file outside of kubernetes and a config map inside. If you interessted I might be able to dig out some pointers (issues)

[cheld]

@floreks Another important reason was that add-on-manager cannot hanlde params properly - just config maps

[floreks]

Updating params stored in ConfigMap and updating Dashboard is basically the same (requires similar steps).

With ConfigMap:

1. Update ConfigMap YAML/Use kubectl edit/Use edit within Dashboard
2. Restart Dashboard to pick up new config

Updating Dashboard:

1. Update Dashboard YAML/Use kubectl edit/Use edit within Dashboard
2. Restart Dashboard/Use kubectl apply to reconfigure Dashboard (this will automatically restart it)

Using kubectl apply actually supports simple versioning because there is option to see last used configuration.

To support this k8s "native" way we'd need to completely redesign the way we read and handle params. Currently they are read only once during Dashboard start and can not be updated without restarting Dashboard. IMO that's ok since they are not changed often. Once you set it and start Dashboard it will run without changes for a long time.

PS. Personally I have never seen any complaints about how we handle params support in Dashboard.

[cheld]

@floreks
The discussion was in core directed at core components like master, controller, etc...

Eventually, we might want t consider and align us with core concepts. But, I agree currently not so relavant for Dashboard.

BTW: I assume it would be possible to avoid the restart with some effort using config map or config file

[Trackhe]

@floreks my dashboard rc1 does not show display metrics. Kubernetes runs on Raspberry pi 4 arm with metrix arm