Querier error : the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed)

[vikasbudhwat]

Describe the bug

Mimir querier giving error when queried for data.
querier ts=2022-11-23T13:12:20.458786731Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.LabelName │ │ s level=warn msg="unable to get store-gateway clients while retrying to fetch missing blocks" err="no store-gateway │ │ instance left after checking exclude for block 01GJJ72920HJX9HR5S56*****" │ │ querier ts=2022-11-23T13:12:20.458815269Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.LabelName │ │ s level=warn user=anonymous msg="failed consistency check" err=null │ │ querier ts=2022-11-23T13:12:20.460855041Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.selectSor │ │ ted level=warn msg="unable to get store-gateway clients while retrying to fetch missing blocks" err="no store-gatewa │ │ y instance left after checking exclude for block 01GJJ72W4YKGC8P2ZE91*****" │ │ querier ts=2022-11-23T13:12:20.460887132Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.selectSor │ │ ted level=warn user=anonymous msg="failed consistency check" err=null

To Reproduce

Steps to reproduce the behavior:
keep the mimir running for enough time till local storage is fulled by header index

1. Start Mimir (SHA or version):
2. Perform Operations(Read/Write/Others): Read

Expected behavior

When queried for metrics/data querier should return the result.

Environment

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
• Deployment tool: [e.g., helm, jsonnet]

Additional Context

Mimir is running on cluster since 2 weeks.
it was tested and querier were fetching the metrics.
mimir config for bucket_store was
bucket_store: sync_dir: /data/tsdb-sync ignore_blocks_within: 5m bucket_index: max_stale_period: 1h

Error logs from querier
querier ts=2022-11-23T13:12:20.458786731Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.LabelName │ │ s level=warn msg="unable to get store-gateway clients while retrying to fetch missing blocks" err="no store-gateway │ │ instance left after checking exclude for block 01GJJ72920HJX9HR5S56QYESCC" │ │ querier ts=2022-11-23T13:12:20.458815269Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.LabelName │ │ s level=warn user=anonymous msg="failed consistency check" err=null │ │ querier ts=2022-11-23T13:12:20.460855041Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.selectSor │ │ ted level=warn msg="unable to get store-gateway clients while retrying to fetch missing blocks" err="no store-gatewa │ │ y instance left after checking exclude for block 01GJJ72W4YKGC8P2ZE91QPGZKJ" │ │ querier ts=2022-11-23T13:12:20.460887132Z caller=spanlogger.go:80 user=anonymous method=blocksStoreQuerier.selectSor │ │ ted level=warn user=anonymous msg="failed consistency check" err=null

post error it was observed that sync-dir(/data/tsdb-sync) was fulled (50Gi), this has header index since few weeks(old one as well).
and the block mentioned in logs are latest blocks(checked in backend storage s3). which are not synced in index-header due to lack of space

tried to set some parameter to clean the header index from locar disk space by configuring
bucket_store: sync_dir: /data/tsdb-sync ignore_blocks_within: 5m index_header_lazy_loading_enabled: true index_header_lazy_loading_idle_timeout: 1d bucket_index: max_stale_period: 1h

but this didn't work .. and no cleanup happens for block index-header.

may be I am missing to parameter to set the block index-header cleanup, timeout to clean local disk. please suggest

[aknuds1]

Converting this to a discussion, since I don't think it identifies a bug.

[pracucci]

I've the feeling there's a misconfiguration. I see you've set ignore_blocks_within: 5m which makes me think you may have changed other advanced config options too, which could conflict with how querying storage work in Mimir. Generally speaking, we recommend to run Mimir with the default configuration and not change advanced parameters, because several config options are coupled together (e.g. if you change config X you should also change config Y accordingly).

That being said, if you could share your entire Mimir config I may take a deeper look.

[vikasbudhwat]

Thanks for response. I have changed the setting as per your suggestion from another thread.
mimir-config.txt

just want to understand as i have setting
bucket_store: ignore_blocks_within: 2h
tsdb: retention_period: 3h
querier: query_store_after: 2h

as bydefault block period would be 2h, so retention period 3h would be enough or not?

BTW do we have any formula for min tsdb block retention value?
with my limited understanding,
min retention_period val = tsdb block_ranges_period + compactor cleanup_interval + few min buffer to upload local compacted block to s3/other long term storage

My understanding

1. compactor compacts and upload local block only after block_ranges_period or when block reached its limit size if any.
2. compactor compacts and upload every cleanup_interval.

[vikasbudhwat]

@pracucci could you please help me with above doubt?

[pracucci]

as bydefault block period would be 2h, so retention period 3h would be enough or not?

It's a stretch. TSDB blocks are 2h ranges, but when a TSDB block is compacted it contains samples from -3h until -1h ago. So I would set retention to at least 4h. However, if you keep reading, you will find out that there are some delays involved until a block is actually queriable. If you want to stay on the safe side, you could set TSDB retention to 5h and query store after 4h.

Also I'm not seeing you to override query_ingesters_within as well. If you set a TSDB retention of 4h in the ingesters, then you should also configure query_ingesters_within: 4h.

BTW do we have any formula for min tsdb block retention value?
with my limited understanding,
min retention_period val = tsdb block_ranges_period + compactor cleanup_interval + few min buffer to upload local compacted > block to s3/other long term storage

min retention_period val should be the sum of the following:

• Time it takes for any ingester to compact TSDB head into a block and upload it to the storage (let's say 10m)
• Time it takes for the compactor cleanup to find the new block and update the bucket index (I would consider it 2x compactor cleanup_interval)
• Time it takes for the store-gateway to update bucket-index, discover new blocks and sync them (I would consider it 2x -blocks-storage.bucket-store.sync-interval). Also if you have memcached enabled, -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl is also a factor to take in account.

querier: query_store_after: 2h

I think you need to give it more time to let store-gateway to discover and sync new blocks.

[vikasbudhwat]

@pracucci thanks for detail explanation.. would change the config as per your above suggestion.

[12nick12]

Where's the URL to the recommend default config?

[pracucci]

Where's the URL to the recommend default config?

We recommend to run Mimir with the built-in default advanced config. The config documentation (and categorization of each configuration between among "basic", "advanced" and "experimental") is here:
https://grafana.com/docs/mimir/latest/operators-guide/configure/reference-configuration-parameters/