[Q&A] Prometheus counter filter not working to emit the metrics for the error/panic/exception in logs

[mukesh-kabra]

What is a problem?

Problem statement: I want to count the number of error/exception/panic substrings present in logs and emit the metrics via Prometheus.
I have tried the below configuration for it to filter the metrics but it's not increasing though the substring is present in the logs

Configuration

  @type tail
  format none
  time_key time
  pos_file /etc/td-agent/loki.pos
  path /log_path
  tag loki.logs
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  read_from_head true
  path_key log_path
</source>

<filter loki.logs>
  @type record_transformer
  enable_ruby
  <record>
    src "#{Socket.gethostname}"
    log_path "${record['log_path']}"
  </record>
</filter>

<filter loki.logs>
  @type prometheus
  <metric>
    name fluentd_input_error_total
    type counter
    desc The total number of error in logs
    key error
    <labels>
      tag ${tag}
      hostname ${hostname}
    </labels>
  </metric>
</filter>

<match loki.logs>
  @type copy
  <store>
    @type loki
    url url
    tenant tenantName
    username username
    password password
    <label>
      src
      log_path
    </label>
    drop_single_key true
    flush_interval 10s
    flush_at_shutdown true
    num_threads 8
    buffer_chunk_limit 1m
    buffer_queue_full_action drop_oldest_chunk
  </store>

  <store>
    @type prometheus
    <metric>
      name fluentd_output_status_num_records_total
      type counter
      desc The total number of outgoing records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </store>

  <store>
    @type prometheus
    <metric>
      name fluentd_output_status_num_error_topics
      type counter
      desc The total number of outgoing records
      key topic
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </store>

</match>


<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

<source>
  @type prometheus_output_monitor
  interval 10
  <labels>
    hostname ${hostname}
  </labels>
</source>

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No logs

Environment

- Fluentd version:
- TD Agent version: 1.11.5
- Operating system: Ubuntu