[Operator] Grafana Agent does not pick Probe's metric relabeling

[MGudin]

Hi! I've deployed the operator alongside a blackbox-exporter on an eks cluster, and i'm scraping blackbox targets thru Probe CRD. That works fine.

What's wrong?

Now i'm trying to add custom labels to each probe via the metricRelabelings config block on probe's CRD but it seems that the agent is ignoring it.

Software version

grafana/agent-operator:v0.39.0

Configuration

Example:

apiVersion: monitoring.grafana.com/v1alpha1
kind: GrafanaAgent
metadata:
  name: blackbox
spec:
  disableReporting: true
  disableSupportBundle: false
  enableConfigReadAPI: false
  metrics:
    externalLabels:
      ...
    instanceSelector:
      matchLabels:
        blackbox: ""
    scrapeInterval: 60s
---
apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  labels:
    blackbox: ""
  name: blackbox
spec:
  probeSelector:
    matchLabels:
      blackbox: ""
  remoteWrite:
  - headers:
      X-Scope-OrgID: ...
    url: ...
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  labels:
    blackbox: ""
  name: myprobe
spec:
  jobName: someprobe
  metricRelabelings:
  - action: replace
    replacement: hello
    targetLabel: mynewlabel
  module: probemodule
  prober:
    path: /probe
    scheme: http
    url: blackbox-exporter:9115
  targets:
    staticConfig:
      static:
      - <target>

What should happen?

Maybe i'm mistaken, but it should create the proper scape config with the relabelConfigs? I checked out the configuration created by the operator and looks like this:

metrics:
    configs:
        - name: <namespace>/blackbox
          remote_write:
            - headers:
                X-Scope-OrgID: ...
              url: ...
          scrape_configs:
            - honor_timestamps: true
              job_name: probe/<namespace>/someprobe
              metrics_path: /probe
              params:
                module:
                    - probemodule
              relabel_configs:
                - source_labels:
                    - job
                  target_label: __tmp_prometheus_job_name
                - replacement: someprobe
                  target_label: job
                - source_labels:
                    - __address__
                  target_label: __param_target
                - source_labels:
                    - __param_target
                  target_label: instance
                - replacement: blackbox-exporter:9115
                  target_label: __address__
              scheme: http
              static_configs:
                - labels:
                    namespace: <namespace>
                  targets:
                    - <target>
    global:
        external_labels:
            __replica__: replica-$(STATEFULSET_ORDINAL_NUMBER)
            ...
        scrape_interval: 60s
    wal_directory: /var/lib/grafana-agent/data
server: {}

I was hoping to see a metric_relabeling_configs block... but it is not showing up. Maybe i'm missing something? Are there any way to achieve what i'm looking for (to add custom labels to metrics via each probe CRD).

Logs

I've checked out operator logs and metrics instance logs and no relevan information there. Just usual reconciles and config reloadings.

Thank's in advance

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!