Failed blocks always after transfer to block storage

[LePau]

I am trying (but failing) to achieve data persistency for mimir across pod restarts (i.e. whether uptime is 3h or 3d).

I have tried both forcing the flush via POST {{MIMIR_URL}}/ingester/flush?wait=true and letting the compaction happen automatically at around 3 hours. Same results either way.

I have tried both defaults and custom values for query_store_after, sync_interval, and cleanup_interval. The end result is always the same.

I am running minio, mimir, and azurite within a minikube instance and minio/azurite PVCs are backed by NFS (nfs-csi, ReadWriteMany).

Any ideas?

Here is the curl command and successful response before data has been sent to block storage:

GET http://localhost:9009/prometheus/api/v1/query_range?query=mtest_endpoint_get_config_count_total&start=2024-06-07T09:03:58.381Z&end=2024-06-09T21:03:58.381Z&step=5m
accept: /
accept-encoding: gzip, deflate, br
user-agent: httpyac

HTTP/1.1 200 - OK
connection: close
content-length: 239
content-type: application/json
date: Sun, 09 Jun 2024 21:03:58 GMT
server-timing: querier_wall_time;dur=8.05872, response_time;dur=3.549595
vary: Accept-Encoding

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"mtest_endpoint_get_config_count_total","instance":"62c9c312-8769-4f59-83ca-82c1dddc0fae","job":"mtest","key":"value"},"values":[[1717966983.381,"15"]]}]}}  
Timings

TCP: 1 ms
First Byte: 12 ms
Download: 2 ms
Total: 15 ms

Meta

ip: 127.0.0.1
size: 421 B

Here is the curl command and response when it errors (after data has been sent to block storage):

GET http://localhost:9009/prometheus/api/v1/query_range?query=mtest_endpoint_get_config_count_total&start=2024-06-07T09:16:58.398Z&end=2024-06-09T21:16:58.398Z&step=5m
accept: /
accept-encoding: gzip, deflate, br
user-agent: httpyac

HTTP/1.1 500 - Internal Server Error
connection: close
content-length: 255
content-type: application/json
date: Sun, 09 Jun 2024 21:16:58 GMT
vary: Accept-Encoding

{"status":"error","errorType":"internal","error":"{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HZZDFCRENDDTR5PWJGREH2Z4\"}"}  
Timings

Wait: 2 ms
DNS: 1 ms
Request: 1 ms
First Byte: 30 ms
Download: 2 ms
Total: 36 ms

Meta

ip: 127.0.0.1
size: 367 B

Here is the error from the mimir log (via kubectl):

ts=2024-06-09T21:16:58.438715868Z caller=spanlogger.go:109 user=anonymous level=error user=anonymous msg="error processing request" try=3 err="{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HZZDFCRENDDTR5PWJGREH2Z4\"}"

ts=2024-06-09T21:16:58.440948629Z caller=spanlogger.go:109 method=blocksStoreQuerier.selectSorted user=anonymous level=warn msg="unable to get store-gateway clients while retrying to fetch missing blocks" err="no store-gateway instance left after checking exclude for block 01HZZDFCRENDDTR5PWJGREH2Z4"

ts=2024-06-09T21:16:58.441014864Z caller=spanlogger.go:109 method=blocksStoreQuerier.selectSorted user=anonymous level=warn user=anonymous msg="failed consistency check after all attempts" err="failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HZZDFCRENDDTR5PWJGREH2Z4"

Edit: I forgot to include something the first time around, and lost the original block ID, but repeated the issue with a new failed block ID and am including the output here. This is the listing on the minio buckets (where the PVC is mounted at /data). The moral of the story here is to show that mimir is indeed writing indexes and blocks into the minio storage - and the error comes later when it tries to read them out. I'm not sure if there is a way to validate integrity of these files.

/data:
total 24
drwxrwsr-x 6 root root 4096 Jun  9 21:07 .
drwxr-xr-x 1 root root 4096 Jun  9 20:57 ..
drwxrwsr-x 7 root root 4096 Jun  9 12:04 .minio.sys
drwxr-sr-x 2 root root 4096 Jun  9 21:07 mimir-alertmanager
drwxr-sr-x 4 root root 4096 Jun 10 00:53 mimir-blocks
drwxr-sr-x 2 root root 4096 Jun  9 21:07 mimir-ruler

/data/.minio.sys:
total 32
drwxrwsr-x 7 root root 4096 Jun  9 12:04 .
drwxrwsr-x 6 root root 4096 Jun  9 21:07 ..
drwxrwsr-x 9 root root 4096 Jun  9 21:07 buckets
drwxrwsr-x 4 root root 4096 Jun  8 13:23 config
-rwxrwxr-- 1 root root  232 Jun  8 13:23 format.json
drwxrwsr-x 2 root root 4096 Jun  8 13:23 multipart
drwxrwsr-x 2 root root 4096 Jun  8 13:23 pool.bin
drwxr-sr-x 3 root root 4096 Jun 10 01:06 tmp

/data/.minio.sys/buckets:
total 36
drwxrwsr-x 9 root root 4096 Jun  9 21:07 .
drwxrwsr-x 7 root root 4096 Jun  9 12:04 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .bloomcycle.bin
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin.bkp
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage.json
drwxr-sr-x 4 root root 4096 Jun  9 21:07 mimir-alertmanager
drwxr-sr-x 4 root root 4096 Jun  9 21:07 mimir-blocks
drwxr-sr-x 4 root root 4096 Jun  9 21:07 mimir-ruler

/data/.minio.sys/buckets/.bloomcycle.bin:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  743 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/.usage-cache.bin:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  677 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/.usage-cache.bin.bkp:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  677 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/.usage.json:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root 3214 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-alertmanager:
total 16
drwxr-sr-x 4 root root 4096 Jun  9 21:07 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin.bkp

/data/.minio.sys/buckets/mimir-alertmanager/.usage-cache.bin:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  587 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-alertmanager/.usage-cache.bin.bkp:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  587 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-blocks:
total 16
drwxr-sr-x 4 root root 4096 Jun  9 21:07 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin.bkp

/data/.minio.sys/buckets/mimir-blocks/.usage-cache.bin:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  667 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-blocks/.usage-cache.bin.bkp:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  667 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-ruler:
total 16
drwxr-sr-x 4 root root 4096 Jun  9 21:07 .
drwxrwsr-x 9 root root 4096 Jun  9 21:07 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .usage-cache.bin.bkp

/data/.minio.sys/buckets/mimir-ruler/.usage-cache.bin:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  580 Jun 10 01:05 xl.meta

/data/.minio.sys/buckets/mimir-ruler/.usage-cache.bin.bkp:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 4 root root 4096 Jun  9 21:07 ..
-rw-r--r-- 1 root root  580 Jun 10 01:05 xl.meta

/data/.minio.sys/config:
total 16
drwxrwsr-x 4 root root 4096 Jun  8 13:23 .
drwxrwsr-x 7 root root 4096 Jun  9 12:04 ..
drwxrwsr-x 2 root root 4096 Jun  8 13:23 config.json
drwxrwsr-x 3 root root 4096 Jun  8 13:23 iam

/data/.minio.sys/config/config.json:
total 20
drwxrwsr-x 2 root root 4096 Jun  8 13:23 .
drwxrwsr-x 4 root root 4096 Jun  8 13:23 ..
-rwxrwxr-- 1 root root 9441 Jun  8 13:23 xl.meta

/data/.minio.sys/config/iam:
total 12
drwxrwsr-x 3 root root 4096 Jun  8 13:23 .
drwxrwsr-x 4 root root 4096 Jun  8 13:23 ..
drwxrwsr-x 2 root root 4096 Jun  8 13:23 format.json

/data/.minio.sys/config/iam/format.json:
total 12
drwxrwsr-x 2 root root 4096 Jun  8 13:23 .
drwxrwsr-x 3 root root 4096 Jun  8 13:23 ..
-rwxrwxr-- 1 root root  434 Jun  8 13:23 xl.meta

/data/.minio.sys/multipart:
total 8
drwxrwsr-x 2 root root 4096 Jun  8 13:23 .
drwxrwsr-x 7 root root 4096 Jun  9 12:04 ..

/data/.minio.sys/pool.bin:
total 12
drwxrwsr-x 2 root root 4096 Jun  8 13:23 .
drwxrwsr-x 7 root root 4096 Jun  9 12:04 ..
-rwxrwxr-- 1 root root  479 Jun  8 13:23 xl.meta

/data/.minio.sys/tmp:
total 16
drwxr-sr-x  3 root root 4096 Jun 10 01:06 .
drwxrwsr-x  7 root root 4096 Jun  9 12:04 ..
drwxr-sr-x 13 root root 4096 Jun 10 01:06 .trash
-rw-r--r--  1 root root 2049 Jun 10 01:05 f49762fb-e5fe-4ed8-b179-ec01adbe5021

/data/.minio.sys/tmp/.trash:
total 52
drwxr-sr-x 13 root root 4096 Jun 10 01:06 .
drwxr-sr-x  3 root root 4096 Jun 10 01:06 ..
drwxr-sr-x  2 root root 4096 Jun 10 01:05 125036da-951d-4f96-89a4-117839ced810
drwxr-sr-x  2 root root 4096 Jun 10 01:06 18c9ad21-7725-4f8c-93ac-038af1dd687d
drwxr-sr-x  2 root root 4096 Jun 10 01:05 24523fdc-8dcf-49f5-b315-383eff759efd
drwxr-sr-x  2 root root 4096 Jun 10 01:05 7c5e6df9-37a2-4476-b6f4-1f4557730999
drwxr-sr-x  2 root root 4096 Jun 10 01:05 7f63a8e8-aa25-48a7-a6e2-60fbead9ae8c
drwxr-sr-x  2 root root 4096 Jun 10 01:05 85a52cc0-6a73-4c88-b421-2573cd99f0ed
drwxr-sr-x  2 root root 4096 Jun 10 01:05 98151580-3f3e-4a75-93ce-3664305c06ba
drwxr-sr-x  2 root root 4096 Jun 10 01:05 9fc23f0b-1d9d-4f1a-8ced-beb5e54824f7
drwxr-sr-x  2 root root 4096 Jun 10 01:05 e99ef018-e4ec-4c2f-94f1-fb0c65b7557e
drwxr-sr-x  2 root root 4096 Jun 10 01:05 eeaf0c83-9a07-4d93-8e58-41196207e56e
drwxr-sr-x  2 root root 4096 Jun 10 01:05 f5f33faa-3728-49a9-9cb8-713803ebb197

/data/.minio.sys/tmp/.trash/125036da-951d-4f96-89a4-117839ced810:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  669 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/18c9ad21-7725-4f8c-93ac-038af1dd687d:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:06 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  718 Jun 10 01:06 xl.meta.bkp

/data/.minio.sys/tmp/.trash/24523fdc-8dcf-49f5-b315-383eff759efd:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  743 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/7c5e6df9-37a2-4476-b6f4-1f4557730999:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  587 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/7f63a8e8-aa25-48a7-a6e2-60fbead9ae8c:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  689 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/85a52cc0-6a73-4c88-b421-2573cd99f0ed:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  580 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/98151580-3f3e-4a75-93ce-3664305c06ba:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root 3215 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/9fc23f0b-1d9d-4f1a-8ced-beb5e54824f7:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  587 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/e99ef018-e4ec-4c2f-94f1-fb0c65b7557e:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  580 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/eeaf0c83-9a07-4d93-8e58-41196207e56e:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  689 Jun 10 01:05 xl.meta.bkp

/data/.minio.sys/tmp/.trash/f5f33faa-3728-49a9-9cb8-713803ebb197:
total 12
drwxr-sr-x  2 root root 4096 Jun 10 01:05 .
drwxr-sr-x 13 root root 4096 Jun 10 01:06 ..
-rw-r--r--  1 root root  669 Jun 10 01:05 xl.meta.bkp

/data/mimir-alertmanager:
total 8
drwxr-sr-x 2 root root 4096 Jun  9 21:07 .
drwxrwsr-x 6 root root 4096 Jun  9 21:07 ..

/data/mimir-blocks:
total 16
drwxr-sr-x 4 root root 4096 Jun 10 00:53 .
drwxrwsr-x 6 root root 4096 Jun  9 21:07 ..
drwxr-sr-x 3 root root 4096 Jun  9 21:54 __mimir_cluster
drwxr-sr-x 5 root root 4096 Jun 10 01:00 anonymous

/data/mimir-blocks/__mimir_cluster:
total 12
drwxr-sr-x 3 root root 4096 Jun  9 21:54 .
drwxr-sr-x 4 root root 4096 Jun 10 00:53 ..
drwxr-sr-x 2 root root 4096 Jun  9 21:54 mimir_cluster_seed.json

/data/mimir-blocks/__mimir_cluster/mimir_cluster_seed.json:
total 12
drwxr-sr-x 2 root root 4096 Jun  9 21:54 .
drwxr-sr-x 3 root root 4096 Jun  9 21:54 ..
-rw-r--r-- 1 root root  521 Jun  9 21:54 xl.meta

/data/mimir-blocks/anonymous:
total 20
drwxr-sr-x 5 root root 4096 Jun 10 01:00 .
drwxr-sr-x 4 root root 4096 Jun 10 00:53 ..
drwxr-sr-x 5 root root 4096 Jun 10 00:53 01HZZT4X75F25KCZPE091KGAME
drwxr-sr-x 5 root root 4096 Jun 10 01:00 01HZZTHQC6Q22PM8FVMVF5F98A
drwxr-sr-x 2 root root 4096 Jun 10 01:06 bucket-index.json.gz

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME:
total 20
drwxr-sr-x 5 root root 4096 Jun 10 00:53 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
drwxr-sr-x 3 root root 4096 Jun 10 00:53 chunks
drwxr-sr-x 3 root root 4096 Jun 10 00:53 index
drwxr-sr-x 2 root root 4096 Jun 10 00:53 meta.json

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/chunks:
total 12
drwxr-sr-x 3 root root 4096 Jun 10 00:53 .
drwxr-sr-x 5 root root 4096 Jun 10 00:53 ..
drwxr-sr-x 3 root root 4096 Jun 10 00:53 000001

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/chunks/000001:
total 16
drwxr-sr-x 3 root root 4096 Jun 10 00:53 .
drwxr-sr-x 3 root root 4096 Jun 10 00:53 ..
drwxr-sr-x 2 root root 4096 Jun 10 00:53 b43e0c2f-d435-4208-955a-9d86dbe47642
-rw-r--r-- 1 root root  365 Jun 10 00:53 xl.meta

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/chunks/000001/b43e0c2f-d435-4208-955a-9d86dbe47642:
total 1296
drwxr-sr-x 2 root root    4096 Jun 10 00:53 .
drwxr-sr-x 3 root root    4096 Jun 10 00:53 ..
-rw-r--r-- 1 root root 1318716 Jun 10 00:53 part.1

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/index:
total 16
drwxr-sr-x 3 root root 4096 Jun 10 00:53 .
drwxr-sr-x 5 root root 4096 Jun 10 00:53 ..
drwxr-sr-x 2 root root 4096 Jun 10 00:53 fca1c56c-8812-48ef-b995-cd9b6913acb7
-rw-r--r-- 1 root root  365 Jun 10 00:53 xl.meta

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/index/fca1c56c-8812-48ef-b995-cd9b6913acb7:
total 3692
drwxr-sr-x 2 root root    4096 Jun 10 00:53 .
drwxr-sr-x 3 root root    4096 Jun 10 00:53 ..
-rw-r--r-- 1 root root 3769892 Jun 10 00:53 part.1

/data/mimir-blocks/anonymous/01HZZT4X75F25KCZPE091KGAME/meta.json:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 00:53 .
drwxr-sr-x 5 root root 4096 Jun 10 00:53 ..
-rw-r--r-- 1 root root 1063 Jun 10 00:53 xl.meta

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A:
total 20
drwxr-sr-x 5 root root 4096 Jun 10 01:00 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
drwxr-sr-x 3 root root 4096 Jun 10 01:00 chunks
drwxr-sr-x 3 root root 4096 Jun 10 01:00 index
drwxr-sr-x 2 root root 4096 Jun 10 01:00 meta.json

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/chunks:
total 12
drwxr-sr-x 3 root root 4096 Jun 10 01:00 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
drwxr-sr-x 3 root root 4096 Jun 10 01:00 000001

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/chunks/000001:
total 16
drwxr-sr-x 3 root root 4096 Jun 10 01:00 .
drwxr-sr-x 3 root root 4096 Jun 10 01:00 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:00 ffabbc9f-1364-4c17-bc5d-14c3add39c43
-rw-r--r-- 1 root root  365 Jun 10 01:00 xl.meta

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/chunks/000001/ffabbc9f-1364-4c17-bc5d-14c3add39c43:
total 7448
drwxr-sr-x 2 root root    4096 Jun 10 01:00 .
drwxr-sr-x 3 root root    4096 Jun 10 01:00 ..
-rw-r--r-- 1 root root 7618442 Jun 10 01:00 part.1

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/index:
total 16
drwxr-sr-x 3 root root 4096 Jun 10 01:00 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
drwxr-sr-x 2 root root 4096 Jun 10 01:00 add1eed2-8bac-4749-8fd3-0d68f7e75635
-rw-r--r-- 1 root root  365 Jun 10 01:00 xl.meta

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/index/add1eed2-8bac-4749-8fd3-0d68f7e75635:
total 3744
drwxr-sr-x 2 root root    4096 Jun 10 01:00 .
drwxr-sr-x 3 root root    4096 Jun 10 01:00 ..
-rw-r--r-- 1 root root 3823720 Jun 10 01:00 part.1

/data/mimir-blocks/anonymous/01HZZTHQC6Q22PM8FVMVF5F98A/meta.json:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:00 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
-rw-r--r-- 1 root root 1064 Jun 10 01:00 xl.meta

/data/mimir-blocks/anonymous/bucket-index.json.gz:
total 12
drwxr-sr-x 2 root root 4096 Jun 10 01:06 .
drwxr-sr-x 5 root root 4096 Jun 10 01:00 ..
-rw-r--r-- 1 root root  718 Jun 10 01:06 xl.meta

/data/mimir-ruler:
total 8
drwxr-sr-x 2 root root 4096 Jun  9 21:07 .
drwxrwsr-x 6 root root 4096 Jun  9 21:07 ..

Here is the mimir config:

common:
  storage:
    backend: s3
    azure:
      account_name: devstoreaccount1
      connection_string: ${AZURE_STORAGE_CONNECTION_STRING}
    s3:
      endpoint: minio:9000
      access_key_id: minioadmin
      secret_access_key: ${S3_OBS_PASSWORD}
      insecure: true

blocks_storage:
  bucket_store:
    # Have tried default and other values
    sync_interval: 15m
  # have tried both azure and s3 (azurite and minio)
  backend: s3
  s3:
    bucket_name: mimir-blocks

alertmanager_storage:
  backend: s3
  s3:
    bucket_name: mimir-alertmanager

ruler_storage:
  backend: s3
  s3:
    bucket_name: mimir-ruler    

multitenancy_enabled: false

compactor:
  # Have tried default and other values
  cleanup_interval: 15m

distributor:
  ring:
    kvstore:
      store: memberlist

ingester:
  ring:
    replication_factor: 1

store_gateway:
  sharding_ring:
    replication_factor: 1

querier:
  # Have tried 3h, 1m, 15m, all the same results
  query_store_after: 10m

server:
  http_listen_port: 9009
  log_level: warn

Here is the kubernetes deployment yaml for mimir:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  generation: 1
  labels:
    app: mimir
  name: mimir
  namespace: default
  resourceVersion: "68614"
spec:
  minReadySeconds: 5
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      application: mimir
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        application: mimir
    spec:
      containers:
      - args:
        - -config.file=/etc/mimir/mimir.yaml
        - -config.expand-env=true
        envFrom:
        - secretRef:
            name: minio-secret
        - secretRef:
            name: azure-secret
        image: grafana/mimir:2.12.0
        imagePullPolicy: IfNotPresent
        name: mimir
        ports:
        - containerPort: 9009
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/mimir
          name: mimir-yaml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: mimir-yaml
        name: mimir-yaml
status:
  availableReplicas: 1
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1