failed to fetch some blocks | err-mimir-store-consistency-check-failed

[JustNZ]

Describe the bug

Mimir is running on 4 dedicated Servers in single-binary mode. After rebooting all 4 Server one after the other to apply some patches, my Mimir servers are in an unstable state. Loading any Grafana Dashboard with the Mimir as DC is returning an "Validation failed" error.

I already saw previous issues about this problem but all of those don't really contain any solution.
I also already tried to remove the blocks from the s3 but after some time the issue reoccurs.

ts=2024-05-08T11:35:40.030943138Z caller=spanlogger.go:109 method=blocksStoreQuerier.selectSorted user=testtrace_id=0c67bd950cf187e2 level=warn user=test traceID=0c67bd950cf187e2 msg="failed consistency check after all attempts" err="failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HXBTBSXVJM3EKPQXXZZEGGG9 01HXBTCNECBGGWXY9W257BQH4G 01HXBTT1VP3DT8NHZ6265CWJW9 01HXBTT5D0QC2NWF5499FD7YN3 01HXBTZ200W6YYTSJN5EZ7F4QR 01HXBTZ64ZAGAN1YPQNFTBBHVN 01HXBV1YM0AX98JHH36KSJ1MX0 01HXBV20T2RJAJ4SK68HYDEG3C"

ts=2024-05-08T11:35:40.030984175Z caller=group.go:503 level=warn trace_id=00000000000000000c67bd950cf187e2 name=MimirCompactorHasNotUploadedBlocks index=49 component=ruler insight=true user=test file=/var/lib/mimir/data/ruler/test/Mimir group=Infra msg="Evaluating rule failed" rule="alert: MimirCompactorHasNotUploadedBlocks\nexpr: (max by (instance, host, type) (thanos_objstore_bucket_last_successful_upload_time{component=\"compactor\",instance=\"MIMIR\"})\n == 0) and (sum by (instance, host, type) (rate(cortex_compactor_group_compaction_runs_started_total{instance=\"MIMIR\"}[1d]))\n > 0)\nfor: 1d\nlabels:\n severity: critical\n team: itops\nannotations:\n description: Mimir Compactor in {{ $labels.instance }}-{{ $labels.stage\n }} {{ $labels.host }} has not uploaded any block since its start.\n summary: Mimir Compactor in {{ $labels.instance }}-{{ $labels.stage\n }} {{ $labels.host }} has not uploaded any block since its start.\n" err="failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HXBTBSXVJM3EKPQXXZZEGGG9 01HXBTCNECBGGWXY9W257BQH4G 01HXBTT1VP3DT8NHZ6265CWJW9 01HXBTT5D0QC2NWF5499FD7YN3 01HXBTZ200W6YYTSJN5EZ7F4QR 01HXBTZ64ZAGAN1YPQNFTBBHVN 01HXBV1YM0AX98JHH36KSJ1MX0 01HXBV20T2RJAJ4SK68HYDEG3C"

To Reproduce

Steps to reproduce the behavior:

1. Restarting one server after the other
2. Wait for all servers to be ready again
3. Access any Dashboard that uses the target Cluster as DC
4. See Error on Dashboard / Logs

Environment

• Infrastructure: RHEL8 Server
• Deployment tool: ansible
• Mimir Version: 2.12.0
• Mode: single-binary

[dimitarvdimitrov]

have you tried following the runbook for this error? https://grafana.com/docs/mimir/latest/manage/mimir-runbooks/#err-mimir-store-consistency-check-failed

[JustNZ]

yes - had a look into that and that was also not resolving my issue.
Was also the first time I see this issue after rebooting all servers running MIMIR

[dimitarvdimitrov]

how do you determine if a server is ready? Do you use the /ready endpoint?

[JustNZ]

yes correct

[dimitarvdimitrov]

If the querier receives an error from the store-gateway it considers the block unqueried. There may be some warning logs from the querier a bit before the consistency check failure log line. If there were errors from the store-gateway, they may be a lead.

Also - are you running with default configuration for -querier.query-store-after and -blocks-storage.bucket-store.sync-interval (and the rest of -querier.* and -blocks-storage.bucket-store.* configs)

[JustNZ]

that is my config for querier and blocks-storage:

blocks_storage:
  storage_prefix: blocks
  bucket_store:
    sync_dir: /var/lib/mimir/data/tsdb-sync/
    ignore_blocks_within: 2h
  tsdb:
    dir: /var/lib/mimir/data/ingester
    flush_blocks_on_shutdown: true

querier:
  query_store_after: 1h

[dimitarvdimitrov]

query_store_after: 1h should be higher than ignore_blocks_within: 2h. Otherwise queriers request blocks which the store-gateway deliberately ignore