metrics not available through fleet-manager endpoint

[k-wall]

Attempts to read the metrics currently fail within the fleet-manager. This occurs because there is no observability component in the kas-installer installed environment.

Internally, fleet-manager fails like this:

E0715 10:17:40.368875       1 api.go:262] error from metric Post "/api/metrics/v1/test/api/v1/query": can't request metrics without auth
E0715 10:17:40.368900       1 metrics.go:55] error getting metrics: KAFKAS-MGMT-9: failed to retrieve metrics

and returns 500 back to the caller.

[k-wall]

Discussions with @pb82 and @JameelB initially had us explored configuring fleetmanager to point directly at the prometheus end-point, however, this doesn't work:

the Observatorium tenant is appended to the URL. the URL Observatorium specific and not compatible with raw Prometheus

https://github.com/pb82/kas-fleet-manager/blob/main/pkg/client/observatorium/client.go#L52

One option is to introduce some special flags into fleet-manager giving it the ability to talk directly to a prometheus. Alternatively, perhaps observatorium could be deployed by the kas-installer.

[k-wall]

@MikeEdgar pointed out to me that kas-installer does have the capability to install observatorium. I'll give this a go and experiment with wiring fleetmanager to it.