getting k8s volume matrices on splunk singnalfx detectors

[jvijayan6]

Hi Team,

I am trying to create new detector for k8s volume usage on signalfx alerts dashboard, but even after adding the below configuration in k8s ClusterReceiver I could see only few matrices are coming not all. values.yaml config is added.

################################################################################
# OpenTelemetry Kubernetes cluster receiver
# This is an extra 1-replica deployment of Open-temlemetry collector used
# specifically for collecting metrics from kubernetes API.
################################################################################

# Cluster receiver collects cluster level metrics from the Kubernetes API.
# It has to be running on one pod, so it uses its own dedicated deployment with 1 replica.

clusterReceiver:
  enabled: true

  # Need to be adjusted based on size of the monitored cluster
  resources:
    limits:
      cpu: 200m
      memory: 500Mi

  # Scheduling configurations
  nodeSelector: {}
  tolerations: []
  affinity: {}

  # Pod configurations
  securityContext: {}
  terminationGracePeriodSeconds: 600
  priorityClassName: ""

  # k8s cluster receiver collector annotations
  annotations: {}
  podAnnotations: {}

  # This flag enables Kubernetes events collection using OpenTelemetry Kubernetes Events Receiver
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver
  # This option requires `logsEnabled` to be set to `true` for either `splunkObservability` or `splunkPlatform`
  # depending on where you want to send the events. Otherwise this option will not have any effect.
  # The receiver currently is in alpha state which means that events format might change over time.
  # Once the receiver is stabilized, it'll be enabled by default in this helm chart
  eventsEnabled: true

  # k8s cluster receiver extra pod labels
  podLabels: {}

  # Extra enviroment variables to be set in the OTel Cluster Receiver container
  extraEnvs: []

  # Extra volumes to be mounted to the k8s cluster receiver container.
  extraVolumes: []
  extraVolumeMounts: []

  # Enable or disable features of the cluster receiver.
  featureGates: ""

  # OpenTelemetry Collector configuration for K8s Cluster Receiver deployment can be overriden in this field.
  # Default configuration defined in templates/config/_otel-k8s-cluster-receiver-config.tpl
  # Any additional fields will be merged into the defaults,
  # existing fields can be disabled by setting them to null value.
  config:
    extensions:
      k8s_observer:
        auth_type: serviceAccount
        observe_pods: true
        observe_nodes: true
    service:
      extensions:
      - health_check
      - k8s_observer
      - memory_ballast
      pipelines:
        metrics:
          exporters:
          - signalfx
          processors:
          - memory_limiter
          - batch
          - resource
          - resource/k8s_cluster
          receivers:
          - k8s_cluster
          - receiver_creator
    receivers:
      receiver_creator:
        receivers:
          kubeletstats:
            rule: type == "k8s.node" && annotations["splunk/scrape"] == "true"
            config:
              auth_type: serviceAccount
              collection_interval: 10s
              endpoint: "${K8S_NODE_NAME}:10250"
              extra_metadata_labels:
                - container.id
                - k8s.volume.type
              k8s_api_config:
                auth_type: serviceAccount
              metric_groups:
                - container
                - pod
                - node
                - volume
        watch_observers:
          - k8s_observer

I am using splunk-otel-collector-chart version 0.67.0
Looking for suggestions...

[jvijayan6]

@dmitryax @emaderer

[jvoravong]

• Why are addiing the kubelet receiver to the clusterReceiver? I think you want to update the agent's kubelet receiver config instead.

  splunk-otel-collector-chart/examples/collector-agent-only/rendered_manifests/configmap-agent.yaml

  Line 165 in 3471875

  - node

• The metrics you are looking for may also be excluded by default.

Here is a sample values file I'd suggest trying.

agent:
  config:
    receivers:
      kubeletstats:
        auth_type: serviceAccount
        collection_interval: 10s
        endpoint: ${K8S_NODE_IP}:10250
        extra_metadata_labels:
        - container.id
        metric_groups:
        - container
        - pod
        - node
        - volume
    exporters:
      signalfx:
        include_metrics:
          # kubelet metrics
          # matches (container|k8s.node|k8s.pod).memory...
          - metric_names:
            # matches k8s.volume.inodes, k8s.volume.inodes and k8s.volume.inodes.used
            - /^k8s\.volume\.inodes(\.free|\.used)*$/

[jvijayan6]

@jvoravong Thanks for the help, added the same and getting some matrices. For testing I have created test PV and PVC, but not getting those metrices. Attaching the screenshot for alert configuration in splunk signalfx.

[jvijayan6]

now the metrices are coming... Thanks a lot for your help