Different query responses on every reload

[David-aylward]

Hi
I'm having an issue with my Mimir, I am very much a noob at using Mimir.
My issue is that with every refresh that Grafana does I'm getting different chunks of data and moving gaps.

I have multiple s3 instances with monolithic Mimir instances connected to each,

I am also using Nginx to load the balance between the 3 instances
The data being written to Mimir is also passing through Nginx and getting load balanced to the various instances.

             [Grafana]       [Writes]
                      \       /
                     [NGINX]
                /      |      \    
       [Mimir 1]  [Mimir 2]  [Mimir 3]
        |            |            |
       [Minion 1] [Minio 2]  [Minio 3]


Config:
blocks_storage:
  backend: s3
  s3:
    endpoint: 192.168.xx.47:9199
    bucket_name: mimir-blocks
    access_key_id: XXXXXX
    secret_access_key: XXXXXX
    insecure: true
##    bucket_store:
  bucket_store:
    bucket_index:
      max_stale_period: 24h      
  tsdb:
    dir: /data/ingester
    ship_concurrency: 100
memberlist:
  join_members: [ 192.168.xx.27, 192.168.xx.15, 192.168.xx.47 ]

ruler:
  rule_path: /data/ruler
  alertmanager_url: http://0.0.0.0:8080/alertmanager
  ring:
    # Quickly detect unhealthy rulers to speed up the tutorial.
    heartbeat_period: 2s
    heartbeat_timeout: 10s

ruler_storage:
#  storage_prefix: blocks
  backend: s3
  s3:
    endpoint: 192.168.xx.47:9199
    bucket_name: mimir-ruler
    access_key_id: XXXXXX
    secret_access_key: XXXXXX
    insecure: true

alertmanager:
  data_dir: /data/alertmanager
  fallback_config_file: /etc/alertmanager/alertmanager-fallback-config.yaml
  external_url: http://localhost:9009/alertmanager

alertmanager_storage:
#  storage_prefix: blocks
  backend: s3
  s3:
    endpoint: 192.168.xx.47:9199
    bucket_name: mimir-alertmanager
    access_key_id: XXXXXX
    secret_access_key: XXXXXX
    insecure: true

server:
  http_listen_port: 9009
  log_level: info
  grpc_server_max_concurrent_streams: 20000
  grpc_server_max_send_msg_size: 1048576000
  grpc_server_max_recv_msg_size: 1048576000
  grpc_listen_address: 0.0.0.0

distributor:
  ring:
    kvstore:
      store: memberlist
  instance_limits:
    max_inflight_push_requests: 0

ingester:
  ring:
    kvstore:
      store: memberlist
    replication_factor: 3
    instance_addr: 0.0.0.0

limits:
  # Delete from storage metrics data older than 1 year.
  compactor_blocks_retention_period: 1y
  max_global_series_per_metric: 0
  out_of_order_time_window: 15m

querier:
  query_store_after: 12h
#  query_ingesters_within: 3h

query_scheduler:
  ring:
    kvstore:
      store: memberlist
compactor:
  cleanup_interval: 15m

[pracucci]

             [Grafana]       [Writes]
                      \       /
                     [NGINX]
                /      |      \    
       [Mimir 1]  [Mimir 2]  [Mimir 3]
        |            |            |
       [Minion 1] [Minio 2]  [Minio 3]

Could you explain what does it mean that each Mimir instance goes to a different Minio? Can you confirm the object storage is shared between the Mimir replicas?

[David-aylward]

The object storage is not shared each mimir instance has its own s3 storage

[replay]

The object storage is not shared each mimir instance has its own s3 storage

This is likely the problem, the object storage should be shared among the Mimir instances