The stateful pods keeps on requiring more disk space.

[sidkram]

Describe the bug

I have loki running on EKS with s3 backend. The performance is splendid and the system is so resilient. But I frequently face a few issues with the stateful components make use of all the disk space. I have frequently updated the disk size of all these components once atleast over a period of 3 weeks. Below is my distribution of statefulsets across different regions and different filesystem sizes. The question here is, 1. based on what factors can I assume the optimal disk size that has to be provisioned to these components 2. At what interval do each of these components do they clean their existing data and pull newer chunks into the filesystem. Probably my second question might look off-point, but I want to understand more in depth on how can I set the alerting right.

|-----------------|-----|----------------|---------|

Component	AZ	Replicas per AZ	Disk Size
Compactor	3AZ	1 replica each	30GB
Ingester	3AZ	2 replicas each	60GB
Store-Gateway	3AZ	1 replica each	50GB
-----------------	-----	----------------	---------

I'm facing the same issue on Loki as well. Is there any configuration parameter that's supposed to be made use of in deciding what should be the optimal disk size. The reason being, I've started at 15 GB but I've already scaled it up to 75 GB over 7-8 iterations. I'd need a possible permanent solution around this.

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version)
2. Perform Operations(Read/Write/Others)

Expected behavior

The stateful pods must be able to go with frequent compaction to eliminate the errors of No space left on the device and run seamlessly

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

mimir-distributed.txt

[aknuds1]

Converting to discussion, as there's no indication of a bug.