filtering.md

Filtering

You may wish to filter out certain datapoints or properties to prevent them from ever leaving the agent. Filtering can be useful to reduce clutter in charts without having to resort to filtering in the UI. If possible, it is preferable to prevent the datapoints and properties you want to omit from being generated by a monitor in the first place, as this will reduce CPU and memory usage of the agent, but sometimes this is not feasible.

Additional monitor-level filtering

It is usually best to apply datapoint filters at the monitor level where the datapoints are being generated. You can do this with the datapointsToExclude option available on every monitor configuration. This option accepts a list of filter items that can specify the metricName(s) and/or dimensions to match on datapoints to determine whether to drop them. For example:

monitors:
   # Prometheus node exporter scraper
 - type: prometheus-exporter
   host: 127.0.0.1
   port: 9090
   datapointsToExclude:
    # This filter causes only the 'free bytes' and 'readonly' filesystem
    # metrics to be sent, excluding all other metrics that start with
    # 'node_filesystem_'.
    - metricNames:
       - node_filesystem_*
       - '!node_filesystem_free_bytes'
       - '!node_filesystem_readonly'

    # This filter causes all network metrics, *except* for those about eth0, to
    # be dropped.
    - metricName: node_network_*
      dimensions:
        interface: [ '*', '!eth0' ]

    # This filter causes all datapoints that start with 'node_disk_' that have
    a 'device' dimension that starts with 'sr' to be dropped.
    - metricName: node_disk_*
      dimensions:
        device: sr*

The behavior is that datapoints are processed through each filter item in the list under datapointsToExclude and if any one of those filters matches, the datapoint is dropped (i.e. the filters are ORed together). Within a single item, for a datapoint to match, it must have a matching metric name (if metricName or metricNames is provided) and any dimensions specified in the filter must match dimensions on the datapoint. There can be extra dimensions on the datapoint that will not affect matching.

The negation of items in either metricNames or a dimensions map value list item will serve to override (and thus reinclude) only other already excluded items from that same list. Thus, if you want to filter metric names or dimension values, you can provide a list like [ '*', '!inclusionList1', '!inclusionList2' ] to exclude everything but those two metrics. This, along with the regex and glob capabilities, is explained more in Overridable Filters.

If there is some monitor configuration option that already prevents the datapoints from being generated in the first place, that is the most ideal, since it reduces CPU consumption by the agent by avoiding the creation of the datapoints in the first place.

Property filtering

Property filtering behaves very similar to datapoint filtering. Filters can be specified with the propertiesToExclude config of the agent config. This parameter should be a list of filters that specify which properties to exclude by defining any or all of the following: dimensionName, dimensionValue, propertyName, propertyValues. Each filter from the list will be applied to all properties the agent emits, so be careful when scoping.

Properties and dimensions (both names and values) can be either globbed (i.e. where * is a wildcard for zero or more characters, and ? is a wildcard for a single character) or specified as a Go-compatible regular expression (the value must be surrounded by / to be considered a regex).

Sometimes it is easier to filter the properties you want to allow through, and not allow any others. You can do this by prefixing any config value with !, which will negate the matching value. This can be applied to any of the config options. See examples below.

Examples:

  # If a property matches any one of these filters it will be excluded
  propertiesToExclude:
    # Exclude 'pod-template-hash' from syncing to any dimension
    - propertyName: pod-template-hash

    # Do not sync any properties to 'kubernetes_pod_uid'
    - dimensionName: kubernetes_pod_uid

    # Exclude 'pod-template-hash'  from syncing on dimension 'kubernetes_pod_uid'
    - propertyName: pod-template-hash
      dimensionName: kubernetes_pod_uid

    # Exclude all properties except 'pod-template-hash' on 'kubernetes_pod_uid'
    -  propertyName: "!pod-template-hash"
       dimensionName: kubernetes_pod_uid

    # Do not sync any property beginning with 'load_balancer' to 'kubernetes_pod_uid'
    - propertyName: "load_balancer*"
      dimensionName: kubernetes_pod_uid

    # Do not sync any property name 'pod-template-hash' beginning with value '123'
    # on a dimension name `kubernetes_pod_uid` with a dimension value starting
    # with 'abc'
    - propertyName: pod-template-hash
      propertyValue: "123*"
      dimensionName: kubernetes_pod_uid
      dimensionValue: "abc*"

Overridable filters

Some config options for monitors and observers support comprehensive filtering that accepts a list of strings that define the filter. These config options support string literals, globbed values, and regular expressions. Filter items can be prefixed by ! to override previously excluded values (which can then also be literal values, globbed values, or regular expressions).

Regular expressions are processed by the Golang regexp package, so the syntax can be anything supported by that. Regular expressions must be surrounded by / on both ends of the string to be interpreted as a regexp. Negation (!) comes before the first / since it is not part of the regexp pattern itself.

Globs can be anything with * (zero or more characters) or ? (zero or one character), along with a few other features. The glob processor library used is demonstrated here.

Examples:

 # Would match the values "a" and "b" and nothing else
 filter:
  - a
  - b

 # Would match all values except for "a"
 filter:
  - "*"
  - "!a"

 # Would match all values matching the pattern disk_* except for "disk_tmp"
 filter:
  - "/disk_.*/"
  - "!disk_tmp"