Mimir kvstore inmemory - crashloops

[med502]

Describe the bug

I'm not able to implement in memory kvstore for ha tracker

Multiple pods are falling into crashloop with error

error loading config from /etc/mimir/mimir.yaml: Error parsing config file: yaml: unmarshal errors:
  line 36: field kvstore not found in type distributor.Config

To Reproduce

Steps to reproduce the behavior:

1. Modiffy default values file with following config

  limits:
    # Adjust max query parallelism to 16x sharding, without sharding we can run 14d queries fully in parallel.
    # With sharding we can further shard each day another 16 times. 14 days * 16 shards = 224 subqueries.
    max_query_parallelism: 224
    max_global_series_per_user: 19000000
    accept_ha_samples: true
    ha_cluster_label: "cluster"
    ha_replica_label: "__replica__"

  distributor:
    ha_tracker:
      enable_ha_tracker: true
      ha_tracker_update_timeout: 15s
      ha_tracker_update_timeout_jitter_max: 5s
      ha_tracker_failover_timeout: 30s
    kvstore:
      store: inmemory
    ring:
      kvstore:
        store: inmemory

2. Implement mimir using helm chart
   helm -n mimir upgrade mimir grafana/mimir-distributed -f mimir-values.yaml

Expected behavior

Mimir should use inmemory kvstore.

Environment

• Infrastructure: Kubernetes EKS cluster
• Deployment tool: helm

Additional Context

I'm trying to implement HA deduplication of metrics which comes from 2 instances of prometheus.
On prometheus side I've configured addtional labels, replicas identifier is based on prometheus hostname as I'm using rancher monitoring stack 2.5
cluster: cluster-dev
replica: $(HOSTNAME)

[pracucci]

👋 Hi! The kvstore should be a child of ha_tracker, so:

  distributor:
    ha_tracker:
      enable_ha_tracker: true
      ha_tracker_update_timeout: 15s
      ha_tracker_update_timeout_jitter_max: 5s
      ha_tracker_failover_timeout: 30s
     # I moved this under ha_tracker 
     kvstore:
        store: inmemory
    ring:
      kvstore:
        store: inmemory

That being said, keep in mind inmemory is intended only for development and shouldn't be used in any real cluster. You should either use memberlist (recommended, except for ha_tracker which is currently unsupported) or consul / etcd.

[med502]

Thanks for suggestion, I will work on consul implementation instead as ha_tracker is not supported yet.