Bug: downstream replied with Unprocessable Entity

[lucasfnds]

Describe the bug

In Grafana, when trying to use Mimir's Datasource, the error appears on the screen: "Validation failed
too many unhealthy instances in the ring". When analyzing the query-frontend logs we can notice:

`ts=2024-08-13T13:47:42.036133953Z caller=handler.go:372 level=info user=anonymous msg="query stats" component=query-frontend method=GET path=/prometheus/api/v1/label/name/values user_agent=Grafana/11.1.3 status_code=422 response_time=2.701379ms response_size_bytes=123 query_wall_time_seconds=0.000694019 fetched_series_count=0 fetched_chunk_bytes=0 fetched_chunks_count=0 fetched_index_bytes=0 sharded_queries=0 split_queries=0 estimated_series_count=0 queue_time_seconds=2.8571e-05 param_start=2024-08-13T12:47:00Z param_end=2024-08-13T13:48:00Z length=1h1m0s time_since_min_time=1h0m42.033364973s time_since_max_time=-17.966635027s results_cache_hit_bytes=0 results_cache_miss_bytes=0 status=failed err="downstream replied with Unprocessable Entity"

ts=2024-08-13T13:47:42.19402017Z caller=handler.go:372 level=info user=anonymous msg="query stats" component=query-frontend method=POST path=/prometheus/api/v1/labels user_agent=Grafana/11.1.3 status_code=422 response_time=3.283817ms response_size_bytes=123 query_wall_time_seconds=0.00154828 fetched_series_count=0 fetched_chunk_bytes=0 fetched_chunks_count=0 fetched_index_bytes=0 sharded_queries=0 split_queries=0 estimated_series_count=0 queue_time_seconds=2.483e-05 param_start=2024-08-13T12:47:00Z param_end=2024-08-13T13:48:00Z length=1h1m0s time_since_min_time=1h0m42.190677582s time_since_max_time=-17.809322418s results_cache_hit_bytes=0 results_cache_miss_bytes=0 status=failed err="downstream replied with Unprocessable Entity"`

I don't have any description of what could be happening, the logs didn't help me.

To Reproduce

Steps to reproduce the behavior:

1. Install grafana mimir using helm;
2. Add the Prometheus Datasource to Grafana pointing to Mimir with Prometheus type" = "Mimir" and save;
3. Click on "Explore view";

Environment

• Infrastructure: Kubernetes
• Deployment tool: Helm
• Mimir helm version: 5.4.0
• Grafana version: 11.1.3

[dimitarvdimitrov]

the querier logs might contain more info

[lucasfnds]

Here are the want logs, they don't say anything after the service has completely initialized, there are some errors but it is before the request and error in Grafana occurs, and nothing more than that appears

querier.log

[Ristovski]

Can confirm, I've hit this as well with the latest grafana and mimir Docker images.