rpc error: code = Unimplemented desc = unknown service cortex.Ingester

[z0rc]

I have multiple prometheus servers that remote write to centralized mimir.

I updated my mimir-distributed helm chart deployment from 3.1.0 to 3.2.0 4 days ago. It was running okay as far as I can tell.

Today though I'm seeing that mimir returns HTTP error 500 on POSTs to /api/v1/push endpoint and distributor logs filled with:

{"caller":"logging.go:86","level":"warn","msg":"POST /api/v1/push (500) 6.636997ms Response: \"rpc error: code = Unimplemented desc = unknown service cortex.Ingester\\n\" ws: false; Accept-Encoding: gzip; Connection: close; C
ontent-Encoding: snappy; Content-Length: 79073; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.39.0; X-Forwarded-For: 10.101.234.178; X-Forwarded-Host: mimir.hjoy.net; X-Forwarded-Port: 443; X-Forwarded-Proto:
 https; X-Forwarded-Server: traefik-private-d97847f65-92gj2; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 10.101.234.178; X-Scope-Orgid: anonymous; ","traceID":"5c60a5ffab53b292","ts":"2022-11-01T18:57:15.363679292Z"}

At the same time all ingester pods are healthy. Logs looks okay to me, except periodic

{"caller":"memberlist_client.go:1225","codec":"usagestats.jsonCodec","key":"collectors/usagestats_token","level":"error","msg":"failed to parse remote state: unknown codec for key","ts":"2022-11-01T19:02:28.914736179Z"}

But this isn't a frequent as remote write POSTs come in.

I also tried to rollback to chart version 3.1.0, but it didn't have any meaningful effect, pushes still return HTTP error 500 and distributor error logs are the same.

I'm struggling to understand what is wrong with my installation, and need directions how to debug this further.

[z0rc]

Okay, I found something. I have tempo running in the same kubernetes cluster as mimir. And as far as I can tell they decided to join same ring, for some reason.

I tried to forget tempo ingesters here, but they come back.

[z0rc]

For the record. I didn't change any default configuration about memberlist/ring in tempo or mimir helm charts. Also they deployed in their own namespaces, and gossip-ring Service resources selectors are sane, they match only pods in their respectful namespaces.

It is still unclear how mimir and tempo were able to join into single ring and what to do with this situation next.

[z0rc]

Seems this situation is similar to grafana/helm-charts#1752.

I updated tempo memberlist configuration, added:

tempo:
  structuredConfig:
    memberlist:
      cluster_label: thanos.tempo
      cluster_label_verification_disabled: false

and restarted tempo via helm upgrade.

It seems fixed the situation. I don't see tempo ingesters in mimir ring anymore and mimir started processing pushes as expected.

[dimitarvdimitrov]

related to #2865

this probably happened because during the rollout mimir reused some of the IPs of tempo pods. So other tempo pods that were running had that IP cached. After some time the tempo pods reached out to that old IP and found a working memberlist node and joined its cluster

The difficulty with enabling this by default in the mimir-distributed chart is that it will require existing installations to go through a migration to avoid temporary downtime.

[pracucci]

For reference, here is the documentation to configure the cluster label verification in Mimir:
https://grafana.com/docs/mimir/latest/operators-guide/configure/configuring-hash-rings/

[z0rc]

Thanks, I also updated mimir configuration per provided instructions.