Autoscaling for ingestors

[giri-thakkellapati]

Hi, I have successfully deployed mimir but I am not able to create auto scaling for mimir ingestors
Is there any option to create auto scaling for mimir ingestors

[pstibrany]

Hello. There is no support for autoscaling ingester yet [*]. We've recently modified our ingester downscaling procedure, and continue to work on autoscaling next.

[*] there is support for autoscaling ingesters when using new Kafka-based ingest-storage with Mimir.

[giri-thakkellapati]

how can we scale down the ingester do we need to enable any config for scaling down

[pstibrany]

Scaling up is easy to do using HPA. Scale downs require more work (see the above link). Procedure from the link is actually implemented in our jsonnet (but not Helm), but we have found issues with it, and we're testing new procedure for downscaling now.

[giri-thakkellapati]

Is it possible to remove the unhealthy instances from the ring

[pstibrany]

It is possible to "forget" such instances on the ring page, accessible at /ingester/ring (distributors and ingesters expose that).

[giri-thakkellapati]

I have a few more question please

1. As i understand, the scaling down doesn't happen immediately as we have the auto scaling based on memory(using HPA), and as the ingesters still hold the memory up to 2 hours. But, will they eventually scale down after this period of 2 hours(assuming the load reduced)?
2. While scaling up, the new pods take time to spin up and register in the distributor. During this time, we see some data is being lost as the existing pods restart and the new pods still coming up while we do the load testing. Is there anyway we can make this process smooth to ensure no data loss?
3. You mentioned Kafka-based ingest-storage with Mimir. above. Can you please share more info on that?
4. Is there a way we can automate forgetting unhealthy ingesters from ring? Probably using unregister_on_shutdown ?

Thanks!

[pstibrany]

We do not support horizontally scaling ingesters based on their memory usage. I cannot help with that, and would not recommend doing it.

You mentioned Kafka-based ingest-storage with Mimir. above. Can you please share more info on that?

This is a new feature we're building into Mimir. In this setup, distributors write data to Kafka, and ingesters read data from Kafka. This separates distributors and ingesters for better reliability. While we're already using this setup in several small clusters inside Grafana Labs, there isn't much public documentation yet, and that's definitely on our TODO list. So far the best documentation is probably in ingest-storage.libsonnet (used to deploy this setup), and some bits and pieces in dashboards (writes.libsonnet, queries.libsonnet, when generated with show_ingest_storage_panels: true).

Is there a way we can automate forgetting unhealthy ingesters from ring? Probably using unregister_on_shutdown ?

We treat unhealthy ingesters in the ring as a sign of problem, that needs to be troubleshooted. Automating forgetting of such entries could lead to data loss or wrong query results, so we don't have any mechanism for it.