Cortex metrics mimir is not displayed

[flowramps]

Cortex_alertmanager_alerts metric is not being displayed in mimir metrics captures.

example of a query being instrumented on my dashboard to show the total alerts

sum(cluster_job_pod:cortex_alertmanager_alerts:sum{cluster=~"$cluster", job=~"($namespace)/((alertmanager|cortex|mimir|mimir-backend.*))"})

Obs: when consulting directly in the prometheus it is also not being displayed

other queries are displayed, example:

version 5.1.4 helm

####values#####
metaMonitoring:
 dashboards:
   annotations:
     k8s-sidecar-target-directory: /tmp/dashboards/Mimir Dashboards
   enabled: true
   labels:
     grafana_dashboard: "1"
 metrics:
   enabled: true
   remote:
     headers: 
       "X-Scope-OrgID": eks-dev|eks-hml
 serviceMonitor:
   enabled: true
   labels:
     release: prometheus
 prometheusRule:
   enabled: true
   mimirAlerts: true
   mimirRules: true
   annotations: {}
   labels:
     release: prometheus
   namespace: mimir-distributed-dev
   groups:
   - name: mimir_dev_alerts #Alertas Inicio
     source_tenants: [ "eks-dev", "eks-hml" ]

[dimitarvdimitrov]

two ideas come to mind

• have you checked on the health of the alertmanager?
• are there any tenants with firing alerts?

[flowramps]

I was analyzing and came across the following questions, thinking about your questions:

• Are there any tenants on fire alerts?

When performing the query, I see that the response returns the value 0 for tenant in the query:

Example of a configured cluster:

cortex_alertmanager_tenants_discovered{cluster="mimir-dev", container="alertmanager", endpoint="http-metrics", instance="10.219.48.163:8080", job="mimir-distributed-dev/alertmanager", namespace="mimir -distributed-dev", pod="mimir-dev-alertmanager-0", service="mimir-dev-alertmanager"}
0

I understand that, in this case, it is not able to identify the tenants and I could not understand how to adjust this.

• Another point: regarding alerts, how could I simulate and configure some tests to generate the grouping and test the query to intentionally catch these groupings?

sum(cluster_job_pod:cortex_alertmanager_alerts:sum{cluster=~"$cluster", job=~"($namespace)/((alertmanager|cortex|mimir|mimir-backend.*))"})

[dimitarvdimitrov]

• Another point: regarding alerts, how could I simulate and configure some tests to generate the grouping and test the query to intentionally catch these groupings?

sum(cluster_job_pod:cortex_alertmanager_alerts:sum{cluster=~"$cluster", job=~"($namespace)/((alertmanager|cortex|mimir|mimir-backend.*))"})

I assume grouping here means series which match cluster_job_pod:cortex_alertmanager_alerts:sum{cluster=~"$cluster", job=~"($namespace)/((alertmanager|cortex|mimir|mimir-backend.*))"}, is that right? You can add an alert rule to the ruler which fires always (for example expr: up{} will always fire if there's a prometheus pushing metrics to mimir). This will create an alert and send it to the alertmanager