Grafana Dashboard showing No Data, GCS Bucket has no Data

[luvpreetsingh]

I have created a separate kubernetes cluster where I have launched prometheus, grafana and mimir using helm. I have used GCS bucket as object storage. Every pod is running fine but the GCS bucket does not have any file or folder in it. The grafana dashboard for mimir writes and mimir reads is also empty. There is no error in any of the pods and I am getting confused now. So, please help me figure out what am I doing wrong here? I will explain my setup below.

Here is prometheus helm values file,

server:

  global:
    scrape_interval: 1m
    scrape_timeout: 30s
    evaluation_interval: 1m
    external_labels:
      cluster: monitoring-cluster

  extraFlags:
    - web.enable-lifecycle
    - web.enable-remote-write-receiver

  remoteWrite:
    - url: http://mimir-nginx/api/v1/push
      remote_timeout: 30s
      follow_redirects: true
      headers:
        X-Scope-OrgID: monitoring-cluster

  persistentVolume:
    size: 120Gi

  statefulSet:
    enabled: true

  retention: "1d"

Here is my mimir helm values file

# this is the mimir config
mimir:
  config: |
    activity_tracker:
      filepath: /data/metrics-activity.log
    blocks_storage:
      backend: gcs
      bucket_store:
        sync_dir: /data/tsdb-sync
      gcs:
        bucket_name: my-gcp-mimir-tsdb
      tsdb:
        dir: /data/tsdb
    compactor:
      data_dir: /data
    frontend:
      align_queries_with_step: true
      log_queries_longer_than: 10s
    frontend_worker:
      frontend_address: mimir-query-frontend-headless.monitoring.svc:9095
    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        unregister_on_shutdown: false
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
    limits:
      max_label_names_per_series: 1000
      max_global_series_per_user: 0
      max_global_series_per_metric: 0
      ingestion_rate: 1000000
      ingestion_burst_size: 2000000
    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - mimir-gossip-ring
    runtime_config:
      file: /var/mimir/runtime.yaml
    server:
      grpc_server_max_concurrent_streams: 1000

minio:
  enabled: false

# don't need alertmanager initially
alertmanager:
  enabled: false

# don't need ruler initially
ruler:
  enabled: false

ingester:
  resources:
    requests:
      cpu: 300m
      memory: 512Mi

  # need high disk space for ingester as it stores the samples on disk
  persistentVolume:
    size: 64Gi

  extraVolumes:
    - name: gcp-creds
      secret:
        secretName: gcp-credentials

  extraVolumeMounts:
    - name: gcp-creds
      mountPath: /var/secrets/google
      readOnly: true

  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/var/secrets/google/gcp-credentials.json"

store_gateway:
  # need high disk space for ingester as it stores fetches the blocks from bucket and stores on the disk
  persistentVolume:
    size: 64Gi

  extraVolumes:
    - name: gcp-creds
      secret:
        secretName: gcp-credentials

  extraVolumeMounts:
    - name: gcp-creds
      mountPath: /var/secrets/google
      readOnly: true

  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/var/secrets/google/gcp-credentials.json"

compactor:
  # need high disk space as it keeps and combines all the blocks
  persistentVolume:
    size: 64Gi

  extraVolumes:
    - name: gcp-creds
      secret:
        secretName: gcp-credentials

  extraVolumeMounts:
    - name: gcp-creds
      mountPath: /var/secrets/google
      readOnly: true

  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/var/secrets/google/gcp-credentials.json"

querier:
  replicas: 2

  resources:
    requests:
      cpu: 300m
      memory: 256Mi

query_frontend:
  replicas: 1

  resources:
    requests:
      cpu: 300m
      memory: 256Mi

There is no helm value file for grafana, I have just launched it as it.

Here is the status of everything running.

$ kubectl get pods -n monitoring
NAME                                            READY   STATUS    RESTARTS   AGE
grafana-85c95778cd-6pqms                        1/1     Running   0          14h
mimir-compactor-0                               1/1     Running   0          28m
mimir-distributor-5b9db4b6c4-crcsv              1/1     Running   0          28m
mimir-ingester-0                                1/1     Running   0          25m
mimir-ingester-1                                1/1     Running   0          26m
mimir-ingester-2                                1/1     Running   0          28m
mimir-nginx-546f7444d9-6tztn                    1/1     Running   0          12h
mimir-overrides-exporter-78b48cbbb-gt2t2        1/1     Running   0          28m
mimir-querier-79d6b7bffc-kbvx4                  1/1     Running   0          28m
mimir-querier-79d6b7bffc-pxhzl                  1/1     Running   0          27m
mimir-query-frontend-7bd69454fd-bqmkf           1/1     Running   0          28m
mimir-store-gateway-0                           1/1     Running   0          28m
prometheus-kube-state-metrics-77ddf69b4-9qk96   1/1     Running   0          24h
prometheus-node-exporter-4r5hm                  1/1     Running   0          14h
prometheus-node-exporter-spgll                  1/1     Running   0          14h
prometheus-server-0                             2/2     Running   0          14h

Everything is running perfectly fine but when I connect grafana to mimir, I get the dashboards showing No Data.

Prometheus is pushing data to mimir and dashboard is also rightly connected to the mimir datasource. I can tell because I see nginx logs. Not going to post all the logs but here it is,

10.64.3.17 - - [30/Jul/2022:08:41:40 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.36.2" "-"
10.64.3.17 - - [30/Jul/2022:08:41:41 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.36.2" "-"
10.64.3.17 - - [30/Jul/2022:08:41:46 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.36.2" "-"
10.64.2.1 - - [30/Jul/2022:08:41:49 +0000]  200 "GET / HTTP/1.1" 2 "-" "kube-probe/1.22" "-"
10.64.2.27 - - [30/Jul/2022:08:41:55 +0000]  200 "POST /prometheus/api/v1/query HTTP/1.1" 63 "-" "Grafana/9.0.4" "-"
10.64.2.27 - - [30/Jul/2022:08:41:55 +0000]  200 "POST /prometheus/api/v1/query HTTP/1.1" 63 "-" "Grafana/9.0.4" "-"
10.64.2.27 - - [30/Jul/2022:08:41:55 +0000]  200 "POST /prometheus/api/v1/query HTTP/1.1" 63 "-" "Grafana/9.0.4" "-"
10.64.2.27 - - [30/Jul/2022:08:41:55 +0000]  200 "POST /prometheus/api/v1/query HTTP/1.1" 63 "-" "Grafana/9.0.4" "-"

Here are the last few logs from prometheus server,

ts=2022-07-30T08:01:37.675Z caller=dedupe.go:112 component=remote level=info remote_name=09f060 url=http://mimir-nginx/api/v1/push msg="Remote storage resharding" from=1 to=2
ts=2022-07-30T08:01:47.675Z caller=dedupe.go:112 component=remote level=info remote_name=09f060 url=http://mimir-nginx/api/v1/push msg="Remote storage resharding" from=2 to=1

So, what am I lacking here? Am I doing anything wrong here? Why is this setup not working?

[pracucci]

Every pod is running fine but the GCS bucket does not have any file or folder in it.

Keep in mind a new Mimir cluster takes about 3 hours before the first blocks are uploaded to the bucket (more recent data only queried from ingesters). I'm mentioning it since I don't know how long Mimir was running.

The grafana dashboard for mimir writes and mimir reads is also empty.

You configured Prometheus with:

      headers:
        X-Scope-OrgID: monitoring-cluster

This means Prometheus is remote writing with the tenant ID monitoring-cluster. When querying back from Mimir you need to inject the same HTTP header, to "authenticate" as monitoring-cluster. Have you configured this HTTP header in the Grafana datasource?

[luvpreetsingh]

When I type cortex in the query builder, it doesn't autocomplete and doesn't show anything. So I think that means that no cortex metrics are there in mimir.

[pracucci]

I had a suspicion if even the mimir data was even being written to mimir?

You need to configure Prometheus to scrape Mimir metrics and then remote write these metrics back to Mimir. You already configured Prometheus to remote write metrics to Mimir, so what's probably missing is enabling Prometheus to scrape Mimir metrics.

Since you're using the Mimir Helm chart, I would recommend following our getting started guide (see doc. In particular, we have a section about metamonitoring, which is basically about scraping Mimir metrics.

[luvpreetsingh]

Thanks. I did that.

Here is the config I added

# enable monitoring of mimir
metaMonitoring:
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true
    installOperator: true
    metrics:
      additionalRemoteWriteConfigs:
        - url: "http://mimir-nginx/api/v1/push"
          headers:
            X-Scope-OrgID: "monitoring-cluster"

But still many panels are empty. Anything else I need to configure in ingester or distributo?

[pracucci]

Anything else I need to configure in ingester or distributo?

No. However, many of our dashboards require recording rules to be configured (either in your Prometheus or Mimir):
https://grafana.com/docs/mimir/latest/operators-guide/monitoring-grafana-mimir/installing-dashboards-and-alerts/#install-from-sources

[luvpreetsingh]

Thanks for all the help @pracucci. Have a good day!