[kube-prometheus-stack] Fix jobLabel on prometheus-node-exporter

[monchaio]

What this PR does / why we need it

When crate prometheus with existing chart I found that jobLabel from metrics of node exporters are not "node-exporter".
After investigation I found that we need to use commonLabels instead of podLabels.
https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-node-exporter/templates/_helpers.tpl#L46.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[jkroepke]

hm. I looked in that PR and I don't understand WHY is solves your issues. I'm unable to find the root cause here.

Normalls, pod and service monitor taking the label from the pod and service I can see that service and pod labels are defined.

[monchaio]

hm. I looked in that PR and I don't understand WHY is solves your issues. I'm unable to find the root cause here.

Normalls, pod and service monitor taking the label from the pod and service I can see that service and pod labels are defined.

Because prometheus-node-exporter change theirs _helper.tpl to use commonLabels instead of podLabels https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-node-exporter/templates/_helpers.tpl#L46

In older version of prometheus-node-exporter they use podLabels.

{{ include "prometheus-node-exporter.selectorLabels" . }}
{{- with .Chart.AppVersion }}
app.kubernetes.io/version: {{ . | quote }}
{{- end }}
{{- with .Values.podLabels }}
{{ toYaml . }}
{{- end }}

Without jobLabel on servicemonitoring. Any metrics from node-exporter have job label in format "<release>-prometheus-node-exporter" and break many rule that still use job label "node-exporter".

node_cpu_seconds_total{mode="idle",job="<release>-prometheus-node-exporter"}

[jkroepke]

If the Service Monitor requires that label, using prometheus.monitor.additionalLabels would enough?

[zeritti]

Without jobLabel on servicemonitoring. Any metrics from node-exporter have job label in format "-prometheus-node-exporter" and break many rule that still use job label "node-exporter".

The label expected by the service monitor, jobLabel, at the Node exporter's service has been set in prometheus-node-exporter.service.labels by default to node-exporter since 62.3.0 (values.yaml). Prometheus operator sets the Node exporter's job name correctly to the label's value.

[jkroepke]

Thats what I observe. The current config should just work. Or I'm wrong?