mimir store-gateway is dropping blocks as outdated (and ignoring blocks retention period)

[r-hempel]

Describe the bug

Chunks are removed from the S3 bucket with an incomprehensible regularity. The compactor was configured in such a way that only blocks older than 180d are expected to be removed.

To Reproduce

• nothing. It is possible that the S3 bucket was unavailable for 5-10 minutes caused by network issues.
• Blocks have been migrated from a prometheus instance TSDB to the S3 bucket (validation was disabled)
• migration happened ~ 7d ago and all blocks were queried by grafana

Expected behavior

Based on the configuration, only blocks older than 180d should be deleted.

Environment

• mimir-distributed helm chart v. 5.2.3 / Kubernetes v1.29.0

• no zone aware replication

• Configuration is as followed

• ` mimir:
  structuredConfig:
  common:
  storage:
  backend: s3
  s3:
  bucket_name: "xxx"
  endpoint: "xxx"

  distributor:        
    extraArgs:
      distributor.ingestion-rate-limit: 35000      
    resources:
      requests:
        cpu: 500m
        memory: 1Gi      
      limits:
        cpu: 1075m
        memory: 1.75Gi
  
  compactor:
    extraArgs:         
      compactor.blocks-retention-period: "180d"
      compactor.block-upload-enabled: true 
      compactor.block-upload-validation-enabled: false
      compactor.block-ranges: 2h0m0s,12h0m0s,24h0m0s
    resources:
      requests:
        cpu: 25m
        memory: 256Mi
      limits:
        cpu: 30m
        memory: 512Mi
    persistentVolume:
      enabled: true
      accessModes:
        - ReadWriteOnce
      size: 8Gi
  
  ingester:
    replicas: 2
    extraArgs:
      ingester.max-global-series-per-user: "0"
    resources:
      requests:
        cpu: 2000m
        memory: 5Gi
      limits:
        cpu: 4000m
        memory: 10.16Gi
    persistentVolume:
      enabled: true
      accessModes:
        - ReadWriteOnce
      size: 10Gi
    zoneAwareReplication:
      enabled: false
  
  store_gateway:
    replicas: 2
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1Gi  
    persistentVolume:
      enabled: true
      accessModes:
        - ReadWriteOnce
      size: 4Gi
    zoneAwareReplication:
      enabled: false      
  
  minio:
    enabled: false
  
  overrides_exporter:
    replicas:  1
    resources:
      requests:
        cpu: 3m
        memory: 21Mi
      limits:
        cpu: 5m
        memory: 50Mi
  
  querier:
    resources:
      requests:
        cpu: 512m
        memory: 512Mi
      limits:
        cpu: 768m
        memory: 768Mi
  
  query_frontend:
    enabled: true
    resources:
      requests:
        cpu: 512m
        memory: 512Mi
      limits:
        cpu: 768m
        memory: 768Mi
  
  query_scheduler:
    enabled: true
    resources:
      requests:
        cpu: 512m
        memory: 512Mi
      limits:
        cpu: 768m
        memory: 768Mi
  
  nginx:
    image:      
      registry: xxx
      repository: xxx
      tag: 1.25-alpine
      client_max_body_size: 1024
    resources:
      requests:
        cpu: 256m
        memory: 256Mi
      limits:
        cpu: 512m
        memory: 512Mi
    podLabels: 
      app: mimir-nginx
  
  rollout_operator:
    image:
      repository: xxx
  
  alertmanager:
    enabled: false
  
  ruler:
    enabled: false
  
  memcached:
    image:      
      repository: xxx    
      tag: xxx
  
  memcachedExporter:
    image:
      repository: xxx
      tag: xxx
  
  chunks-cache:
    enabled: true
  
  index-cache:
    enabled: true
  
  metadata-cache:
    enabled: true
  
  metaMonitoring:
    dashboards:
      enabled: false
  
    serviceMonitor:
      enabled: true
      interval: 1m
  
    prometheusRule:
      enabled: true
      mimirAlerts: true
      mimirRules: true
      namespace: observability`
  

Additional Context

Logs in store-gateway:
ts=2024-08-06T08:09:38.13164158Z caller=bucket.go:405 level=info user=anonymous msg="loaded new block" elapsed=464.109706ms id=01J3HZH7R5DGA08X557KWF9CFX ts=2024-08-06T08:09:38.132932997Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J1R8H0AGNAY25ECPESWTNP39 ts=2024-08-06T08:09:38.134626416Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J1JF4DZTNVXXNB2JPEPQ3BSA ts=2024-08-06T08:09:38.136119615Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J23V9DXMFWWWN833TPD85MXS ts=2024-08-06T08:09:38.137688299Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J29MQ86ACQMCXDYAN7EF38MP ts=2024-08-06T08:09:38.138515917Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J3YB6ER9HZBGHD5A6DCYS1Z4 ts=2024-08-06T08:09:38.140130299Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J1Y1Y2PWVN2CG4CH07KBMAS7 ts=2024-08-06T08:09:38.140878692Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J40T4P6QCTF3MGKG8TH87SVH

• currently there are three tenants in mimir, but only one is used.

Thanks for the support in advance!

[r-hempel]

Hi,

I did some further research and found the following logs after it happened again:

Mimir logs are as following:

ts=2024-09-06T10:22:53.874762874Z caller=blocks_cleaner.go:195 level=info component=cleaner run_id=1725618173 task=clean_up_users msg="started blocks cleanup and maintenance"
ts=2024-09-06T10:22:54.343580153Z caller=blocks_cleaner.go:389 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="started blocks cleanup and maintenance"
ts=2024-09-06T10:22:54.359368581Z caller=blocks_cleaner.go:406 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="fetched existing bucket index"
ts=2024-09-06T10:22:54.35942324Z caller=blocks_cleaner.go:586 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="marked blocks for deletion" num_blocks=0 retention=4320h0m0s
ts=2024-09-06T10:22:54.455172709Z caller=updater.go:99 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="listed all blocks in storage" newly_discovered=0 existing=49
ts=2024-09-06T10:22:54.455214984Z caller=updater.go:123 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="fetched blocks metas for newly discovered blocks" total_blocks=28 partial_errors=0
ts=2024-09-06T10:22:54.457831031Z caller=updater.go:186 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="listed deletion markers" count=0
ts=2024-09-06T10:22:54.457869987Z caller=updater.go:215 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0
ts=2024-09-06T10:22:54.466382655Z caller=blocks_cleaner.go:394 level=info component=cleaner run_id=1725618173 task=clean_up_users user=anonymous msg="completed blocks cleanup and maintenance" duration=122.801669ms
ts=2024-09-06T10:22:54.466449734Z caller=blocks_cleaner.go:201 level=info component=cleaner run_id=1725618173 task=clean_up_users msg="successfully completed blocks cleanup and maintenance"

Object store are as following:

ts=2024-09-06T10:29:35.396577654Z caller=gateway.go:307 level=info msg="synchronizing TSDB blocks for all users" reason=periodic
ts=2024-09-06T10:29:35.437163565Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J56VW5TRSY2CE5SQS84MGFWZ
ts=2024-09-06T10:29:35.438322151Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J52C4HZPS90T136GM1SPXCCW
ts=2024-09-06T10:29:35.439628901Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J2V0X53EXG91JJ49937VM0KT
ts=2024-09-06T10:29:35.440563357Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J30T9G5J6YNBFKHQ9SA1453P
ts=2024-09-06T10:29:35.441988612Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J2FE3YZJFFH1RGWCXJJ4CCN8
ts=2024-09-06T10:29:35.442880585Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J3CD27NJTMA4W6WN0VGW3HE9
ts=2024-09-06T10:29:35.444175178Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J6GSWB0NXY100DBGHT5KTTNG
ts=2024-09-06T10:29:35.44486938Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J462HH8V8VDQN3DSAJRZ2VDV
ts=2024-09-06T10:29:35.446204585Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J1Y1Y2PWVN2CG4CH07KBMAS7
ts=2024-09-06T10:29:35.446864854Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J4B7SM0WRPPVR62G9MNDQR5E
ts=2024-09-06T10:29:35.44821008Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J23V9DXMFWWWN833TPD85MXS
ts=2024-09-06T10:29:35.450206863Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J4DQ7VW8YB4FEZ0G2YW8NGSA
ts=2024-09-06T10:29:35.450878365Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J6E5HD9JGCDCJ1TDC12543MT
ts=2024-09-06T10:29:35.452265353Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J29MQ86ACQMCXDYAN7EF38MP
ts=2024-09-06T10:29:35.453229283Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J36KNXTJXRCAQP55VKXM9CNC
ts=2024-09-06T10:29:35.454686771Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J48MEDAZXCDC1PMA7A2T4Q66
ts=2024-09-06T10:29:35.455482089Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J6BJKA4MRKQSQW8WQQE0DZ2F
ts=2024-09-06T10:29:35.456110869Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J4G9AQQWMK56CAD4KB39DR5X
ts=2024-09-06T10:29:35.457244304Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J57NFSNCNSYC4A60QZJDP8TF
ts=2024-09-06T10:29:35.457872834Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J5HPY2JBTV7DV5J4N7XGSM9E
ts=2024-09-06T10:29:35.459037275Z caller=bucket.go:347 level=info user=anonymous msg="dropped outdated block" block=01J2N7GCY5YYNEJ7BAC1NR78TB
ts=2024-09-06T10:29:35.459147368Z caller=gateway.go:313 level=info msg="successfully synchronized TSDB blocks for all users" reason=periodic

So I think the compactor is somehow failing in updating the block index (but not everytime). The pods are monitored and there is nothing special in resource consumption at the moment of deletion. I checked the source code and I can not explain, how the index-entries are getting lost.

Any idea, anyone? :-D