How to configure Prometheus exporter in Kubernetes?

[g0loob]

Hello 👋

Components

exporter/prometheusexporter

Problem

I can't see metrics from the Collector in the Prometheus UI. I suspect that my configuration is wrong, making Prometheus unable to scrape them, but I can't figure out what's wrong.

Setup

I'm using:

• OpenTelemetry Collector Helm chart
• kube-prometheus-stack Helm chart
• Jaeger Helm chart

to create a simple example of receiving OTLP telemetry data and export it to Jaeger and Prometheus.

I have a following override of values.yaml file for the Collector:

mode: "deployment"

fullnameOverride: "otel-collector"

presets:
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
  
  exporters:
    debug:
      verbosity: detailed
    
    otlp/jaeger:
      endpoint: jaeger-collector.default.svc.cluster.local:14250
      tls:
        insecure: true

    prometheus:
      endpoint: ${env:MY_POD_IP}:8889
  
  service:
    pipelines:
      traces:
        receivers:
          - otlp
        processors:
          - memory_limiter
          - batch
        exporters:
          - otlp/jaeger
      
      metrics:
        receivers:
          - otlp
          - prometheus
        processors:
          - memory_limiter
          - batch
        exporters:
          - debug
          - prometheus
      
      logs: null

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
  metrics:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

resources:
  limits:
    cpu: 2
    memory: 4Gi

ingress:
  enabled: false

serviceMonitor:
  enabled: true

Expectation

Once, the metrics port (8888) and the ServiceMonitor are enabled, I expected that Prometheus will be abled to scrape Collector's metrics. Unfortunately, I can't find any metric from the debug exporter in the Prometheus UI.

Additional information

• I'm avoiding using remote write receiver as it's an experimental feature.
• I'm avoiding using OTLP receiver as it's an experimental feature.
• When I port-forward metrics port 8888, I'm not able to reach localhost:8888/metrics endpoint and get Prometheus metrics.

---

Could you provide an example how to configure Collector with Prometheus? Or point out the mistakes in my configuration?

I managed to do it with "plain" Docker containers, but I'm having hard times configuring Kubernetes resources.

[g0loob]

The problem was that I was unaware that Prometheus CRD has a ServiceMonitor selector. By default, it only looked for ServiceMonitors which had release: prometheus label. After changing the configuration of ServiceMonitor to:

serviceMonitor:
  enabled: true

  extraLabels:
    release: prometheus

I was able to see the metrics in the UI.