bug: writing block: closing index writer: postings offset table size exceeds 4 bytes: 5078829125

[shichanglin5]

compactor error log:

writing block: closing index writer: postings offset table size exceeds 4 bytes: 5078829125

ts=2024-03-23T17:37:08.20369681Z caller=bucket_compactor.go:260 level=error component=compactor user=skyeye-apm groupKey=0@17241709254077376921-merge--1710374400000-1710460800000 minTime="2024-03-14 00:00:00 +0000 UTC" maxTime="2024-03-15 00:00:00 +0000 UTC" msg="compaction job failed" duration=2h28m58.254763326s duration_ms=8938254 err="compact blocks [/data/compact/0@17241709254077376921-merge--1710374400000-1710460800000/01HSBGWJQK0KAMMY9NQX46M37Z /data/compact/0@17241709254077376921-merge--1710374400000-1710460800000/01HSAZSHC8M8GGYJM2RXSRGAD9]: writing block: closing index writer: postings offset table size exceeds 4 bytes: 5078829125"

[aknuds1]

Can you give us the following parameters please?

• The version of Mimir
• The infrastructure (Kubernetes?)
• The deployment tool (Helm or Jsonnet?)

Do you have any tips on reproducing the problem?

[shichanglin5]

Can you give us the following parameters please?

• The version of Mimir
• The infrastructure (Kubernetes?)
• The deployment tool (Helm or Jsonnet?)

Do you have any tips on reproducing the problem?

mimir git commit revision: 71ff092

I use Helm to deploy mimir on k8s

[pstibrany]

postings offset table size exceeds 4 bytes: 5078829125

This to me is an indication that your block index is too big, and you will need to configure series sharding in the compactor, so that series are stored into multiple blocks. See https://grafana.com/docs/mimir/latest/references/architecture/components/compactor/#compaction-algorithm for details, and especially -compactor.split-and-merge-shards parameter.

[aknuds1]

Converting to a discussion, since it shouldn't be an actual bug. See @pstibrany's answer above.

[aknuds1]

@pstibrany also notes that a potential improvement would be if Mimir could detect this scenario, and provide a more helpful error message.