v0.37.0-rc.1

This is release v0.37.0-rc.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

This second release candidate fixes a bug affecting the node_exporter integration and the prometheus.exporter.unix component.

The biggest news is that Flow mode has now feature parity with Static mode! That means that you can use Flow components for all static mode use cases.

The v0.37.0-rc.0 release contains some breaking changes:

Breaking changes

• Set retry_on_http_429 to true by default in the queue_config block in static mode's remote_write. (@wildum)
• Renamed non_indexed_labels Loki processing stage to structured_metadata. (@vlad-diachenko)
• Include otel_scope_name and otel_scope_version in all metrics for otelcol.exporter.prometheus
  by default using a new argument include_scope_labels. (@erikbaranowski)
• Static mode Windows Certificate Filter no longer restricted to TLS 1.2 and specific cipher suites. (@mattdurham)
• The __meta_agent_integration* and __meta_agent_hostname labels have been
  removed from the targets exposed by prometheus.exporter.* components and
  got replaced by the pair of __meta_component_name and __meta_component_id
  labels. (@tpaschalis)
• Flow: Allow prometheus.exporter.unix to be specified multiple times and used in modules. This now means all
  prometheus.exporter.unix references will need a label prometheus.exporter.unix "example". (@mattdurham)

Also, it includes a number of new features, including 23 (!) new Flow components, improved evaluation of the Flow component graph, new config converters as well as the ability to run multiple River files at once.

Features

• New Grafana Agent Flow components:

  • discovery.consulagent discovers scrape targets from Consul Agent. (@wildum)
  • discovery.kuma discovers scrape targets from the Kuma control plane. (@tpaschalis)
  • discovery.linode discovers scrape targets from the Linode API. (@captncraig)
  • discovery.marathon discovers scrape targets from Marathon servers. (@wildum)
  • discovery.ionos discovers scrape targets from the IONOS Cloud API. (@wildum)
  • discovery.triton discovers scrape targets from Triton Container Monitor. (@erikbaranowski)
  • discovery.nerve discovers scrape targets from AirBnB's Nerve. (@tpaschalis)
  • discovery.serverset discovers Serversets stored in Zookeeper. (@thampiotr)
  • discovery.scaleway discovers scrape targets from Scaleway virtual
    instances and bare-metal machines. (@rfratto)
  • faro.receiver accepts Grafana Faro-formatted telemetry data over the
    network and forwards it to other components. (@megumish, @rfratto)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • discovery.dockerswarm discovers scrape targets from Docker Swarm. (@wildum)
  • otelcol.connector.servicegraph creates service graph metrics from spans. It is the
    flow mode equivalent to static mode's service_graphs processor. (@ptodev)
  • otelcol.connector.spanlogs creates logs from spans. It is the flow mode equivalent
    to static mode's automatic_logging processor. (@ptodev)
  • otelcol.processor.k8sattributes adds Kubernetes metadata as resource attributes
    to spans, logs, and metrics. (@acr92)
  • otelcol.processor.probabilistic_sampler samples logs and traces based on configuration options. (@mar4uk)
  • otelcol.processor.transform transforms OTLP telemetry data using the
    OpenTelemetry Transformation Language (OTTL). It is most commonly used
    for transformations on attributes.
  • remote.kubernetes.configmap loads a configmap's data for use in other components (@captncraig)
  • remote.kubernetes.secret loads a secret's data for use in other components (@captncraig)
  • prometheus.exporter.agent exposes the agent's internal metrics. (@hainenber)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • prometheus.exporter.cadvisor exposes cAdvisor metrics. (@tpaschalis)
  • prometheus.exporter.vsphere exposes vmware vsphere metrics. (@marctc)

• Flow: allow the HTTP server to be configured with TLS in the config file
  using the new http config block. (@rfratto)

• Clustering: add new flag --cluster.max-join-peers to limit the number of peers the system joins. (@wildum)

• Clustering: add a new flag --cluster.name to prevent nodes without this identifier from joining the cluster. (@wildum)

• Clustering: add IPv6 support when using advertise interfaces to assign IP addresses. (@wildum)

• Add a file_watch block in loki.source.file to configure how often to poll files from disk for changes via min_poll_frequency and max_poll_frequency.
  In static mode it can be configured in the global file_watch_config via min_poll_frequency and max_poll_frequency. (@wildum)

• Flow: In prometheus.exporter.blackbox, allow setting labels for individual targets. (@spartan0x117)

• Add optional nil_to_zero config flag for YACE which can be set in the static, discovery, or metric config blocks. (@berler)

• The cri stage in loki.process can now be configured to limit line size.

• Flow: Allow grafana-agent run to accept a path to a directory of *.river files.
  This will load all River files in the directory as a single configuration;
  component names must be unique across all loaded files. (@rfratto, @hainenber)

• Added support for static configuration conversion in grafana-agent convert and grafana-agent run commands. (@erikbaranowski)

• Flow: the prometheus.scrape component can now configure the scraping of
  Prometheus native histograms. (@tpaschalis)

• Flow: the prometheus.remote_write component now supports SigV4 and AzureAD authentication. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.37.0-rc.0

This is release v0.37.0-rc.0 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

The biggest news is that Flow mode has now feature parity with Static mode! That means that you can use Flow components for all static mode use cases.

The v0.37.0-rc.0 release contains some breaking changes:

Breaking changes

• Set retry_on_http_429 to true by default in the queue_config block in static mode's remote_write. (@wildum)
• Renamed non_indexed_labels Loki processing stage to structured_metadata. (@vlad-diachenko)
• Include otel_scope_name and otel_scope_version in all metrics for otelcol.exporter.prometheus
  by default using a new argument include_scope_labels. (@erikbaranowski)
• Static mode Windows Certificate Filter no longer restricted to TLS 1.2 and specific cipher suites. (@mattdurham)
• The __meta_agent_integration* and __meta_agent_hostname labels have been
  removed from the targets exposed by prometheus.exporter.* components and
  got replaced by the pair of __meta_component_name and __meta_component_id
  labels. (@tpaschalis)
• Flow: Allow prometheus.exporter.unix to be specified multiple times and used in modules. This now means all
  prometheus.exporter.unix references will need a label prometheus.exporter.unix "example". (@mattdurham)

Also, it includes a number of new features, including 23 (!) new Flow components, improved evaluation of the Flow component graph, new config converters as well as the ability to run multiple River files at once.

Features

• New Grafana Agent Flow components:

  • discovery.consulagent discovers scrape targets from Consul Agent. (@wildum)
  • discovery.kuma discovers scrape targets from the Kuma control plane. (@tpaschalis)
  • discovery.linode discovers scrape targets from the Linode API. (@captncraig)
  • discovery.marathon discovers scrape targets from Marathon servers. (@wildum)
  • discovery.ionos discovers scrape targets from the IONOS Cloud API. (@wildum)
  • discovery.triton discovers scrape targets from Triton Container Monitor. (@erikbaranowski)
  • discovery.nerve discovers scrape targets from AirBnB's Nerve. (@tpaschalis)
  • discovery.serverset discovers Serversets stored in Zookeeper. (@thampiotr)
  • discovery.scaleway discovers scrape targets from Scaleway virtual
    instances and bare-metal machines. (@rfratto)
  • faro.receiver accepts Grafana Faro-formatted telemetry data over the
    network and forwards it to other components. (@megumish, @rfratto)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • discovery.dockerswarm discovers scrape targets from Docker Swarm. (@wildum)
  • otelcol.connector.servicegraph creates service graph metrics from spans. It is the
    flow mode equivalent to static mode's service_graphs processor. (@ptodev)
  • otelcol.connector.spanlogs creates logs from spans. It is the flow mode equivalent
    to static mode's automatic_logging processor. (@ptodev)
  • otelcol.processor.k8sattributes adds Kubernetes metadata as resource attributes
    to spans, logs, and metrics. (@acr92)
  • otelcol.processor.probabilistic_sampler samples logs and traces based on configuration options. (@mar4uk)
  • otelcol.processor.transform transforms OTLP telemetry data using the
    OpenTelemetry Transformation Language (OTTL). It is most commonly used
    for transformations on attributes.
  • remote.kubernetes.configmap loads a configmap's data for use in other components (@captncraig)
  • remote.kubernetes.secret loads a secret's data for use in other components (@captncraig)
  • prometheus.exporter.agent exposes the agent's internal metrics. (@hainenber)
  • prometheus.exporter.azure collects metrics from Azure. (@wildum)
  • prometheus.exporter.cadvisor exposes cAdvisor metrics. (@tpaschalis)
  • prometheus.exporter.vsphere exposes vmware vsphere metrics. (@marctc)

• Flow: allow the HTTP server to be configured with TLS in the config file
  using the new http config block. (@rfratto)

• Clustering: add new flag --cluster.max-join-peers to limit the number of peers the system joins. (@wildum)

• Clustering: add a new flag --cluster.name to prevent nodes without this identifier from joining the cluster. (@wildum)

• Clustering: add IPv6 support when using advertise interfaces to assign IP addresses. (@wildum)

• Add a file_watch block in loki.source.file to configure how often to poll files from disk for changes via min_poll_frequency and max_poll_frequency.
  In static mode it can be configured in the global file_watch_config via min_poll_frequency and max_poll_frequency. (@wildum)

• Flow: In prometheus.exporter.blackbox, allow setting labels for individual targets. (@spartan0x117)

• Add optional nil_to_zero config flag for YACE which can be set in the static, discovery, or metric config blocks. (@berler)

• The cri stage in loki.process can now be configured to limit line size.

• Flow: Allow grafana-agent run to accept a path to a directory of *.river files.
  This will load all River files in the directory as a single configuration;
  component names must be unique across all loaded files. (@rfratto, @hainenber)

• Added support for static configuration conversion in grafana-agent convert and grafana-agent run commands. (@erikbaranowski)

• Flow: the prometheus.scrape component can now configure the scraping of
  Prometheus native histograms. (@tpaschalis)

• Flow: the prometheus.remote_write component now supports SigV4 and AzureAD authentication. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.36.2

This is release v0.36.2 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Bugfixes

• Fixed a bug where otelcol.processor.discovery could modify the targets passed by an upstream component. (@ptodev)

• Fixed a bug where otelcol components with a retry mechanism would not wait after the first retry. (@rfratto)

• Fixed a bug where documented default settings in otelcol.exporter.loadbalancing were never set. (@rfratto)

• Fix loki.source.file race condition in cleaning up metrics when stopping to tail files. (@thampiotr)

• Fixed the agent_prometheus_scrape_targets_gauge incorrectly reporting all discovered targets
  instead of targets that belong to current instance when clustering is enabled. (@thampiotr)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.36.1

This is release v0.36.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

• Restart managed components of a module loader only on if module content changes or the last load failed. This was specifically impacting module.git each time it pulls. (@erikbaranowski)
• Allow overriding default User-Agent for http.remote component (@hainenber)
• Fix panic when running grafana-agentctl config-check against config files having integrations block (both V1 and V2). (@hainenber)
• Fix a deadlock candidate in the loki.process component. (@tpaschalis)
• Fix an issue in the eventhandler integration where events would be double-logged: once by sending the event to Loki, and once by including the event in the Grafana Agent logs. Now, events are only ever sent to Loki. (@rfratto)
• Converters will now sanitize labels to valid River identifiers. (@erikbaranowski)
• Fix an issue in converters where targets of discovery.relabel components were repeating the first target for each source target instead of the correct target. (@erikbaranowski)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.36.0

This is release v0.36.0 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Breaking changes

• loki.source.file component will no longer automatically detect and
  decompress logs from compressed files. A new configuration block is available
  to enable decompression explicitly. See the [upgrade guide][] for migration
  instructions. (@thampiotr)

• otelcol.exporter.prometheus: Set include_scope_info to false by default. You can set
  it to true to preserve previous behavior. (@gouthamve)

• prometheus.remote_write: Set retry_on_http_429 to true by default in the queue_config block.
  You can set it to false to preserve previous behavior. (@wildum)

Features

• Add godeltaprof profiling types (godeltaprof_memory, godeltaprof_mutex, godeltaprof_block) to pyroscope.scrape component

• Flow: Allow the logging configuration block to tee the Agent's logs to one
  or more loki.* components. (@tpaschalis)

• Added support for promtail configuration conversion in grafana-agent convert and grafana-agent run commands. (@thampiotr)

• Flow: Add a new stage non_indexed_labels to attach non-indexed labels from extracted data to log line entry. (@vlad-diachenko)

• loki.write now exposes basic WAL support. (@thepalbi)

• Flow: Users can now define additional_fields in loki.source.cloudflare (@wildum)

• Flow: Added exemplar support for the otelcol.exporter.prometheus. (@wildum)

• Add a labels argument in loki.source.windowsevent to associate additional labels with incoming logs. (@wildum)

• New Grafana Agent Flow components:

  • prometheus.exporter.gcp - scrape GCP metrics. (@tburgessdev)
  • otelcol.processor.span - accepts traces telemetry data from other otelcol
    components and modifies the names and attributes of the spans. (@ptodev)
  • discovery.uyuni discovers scrape targets from a Uyuni Server. (@sparta0x117)
  • discovery.eureka discovers targets from a Eureka Service Registry. (@spartan0x117)
  • discovery.openstack - service discovery for OpenStack. (@marctc)
  • discovery.hetzner - service discovery for Hetzner Cloud. (@marctc)
  • discovery.nomad - service discovery from Nomad. (@captncraig)
  • discovery.puppetdb - service discovery from PuppetDB. (@captncraig)
  • otelcol.processor.discovery adds resource attributes to spans, where the attributes
    keys and values are sourced from discovery.* components. (@ptodev)
  • otelcol.connector.spanmetrics - creates OpenTelemetry metrics from traces. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.36.0-rc.3

This is release v0.36.0-rc.3 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Breaking changes

• loki.source.file component will no longer automatically detect and
  decompress logs from compressed files. A new configuration block is available
  to enable decompression explicitly. See the upgrade guide for migration
  instructions. (@thampiotr)

• otelcol.exporter.prometheus: Set include_scope_info to false by default. You can set
  it to true to preserve previous behavior. (@gouthamve)

• prometheus.remote_write: Set retry_on_http_429 to true by default in the queue_config block.
  You can set it to false to preserve previous behavior. (@wildum)

Features

• Add godeltaprof profiling types (godeltaprof_memory, godeltaprof_mutex, godeltaprof_block) to pyroscope.scrape component

• Flow: Allow the logging configuration block to tee the Agent's logs to one
  or more loki.* components. (@tpaschalis)

• Added support for promtail configuration conversion in grafana-agent convert and grafana-agent run commands. (@thampiotr)

• Flow: Add a new stage non_indexed_labels to attach non-indexed labels from extracted data to log line entry. (@vlad-diachenko)

• loki.write now exposes basic WAL support. (@thepalbi)

• Flow: Users can now define additional_fields in loki.source.cloudflare (@wildum)

• Flow: Added exemplar support for the otelcol.exporter.prometheus. (@wildum)

• Add a labels argument in loki.source.windowsevent to associate additional labels with incoming logs. (@wildum)

• New Grafana Agent Flow components:

  • prometheus.exporter.gcp - scrape GCP metrics. (@tburgessdev)
  • otelcol.processor.span - accepts traces telemetry data from other otelcol
    components and modifies the names and attributes of the spans. (@ptodev)
  • discovery.uyuni discovers scrape targets from a Uyuni Server. (@sparta0x117)
  • discovery.eureka discovers targets from a Eureka Service Registry. (@spartan0x117)
  • discovery.openstack - service discovery for OpenStack. (@marctc)
  • discovery.hetzner - service discovery for Hetzner Cloud. (@marctc)
  • discovery.nomad - service discovery from Nomad. (@captncraig)
  • discovery.puppetdb - service discovery from PuppetDB. (@captncraig)
  • otelcol.processor.discovery adds resource attributes to spans, where the attributes
    keys and values are sourced from discovery.* components. (@ptodev)
  • otelcol.connector.spanmetrics - creates OpenTelemetry metrics from traces. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.35.4

This is release v0.35.4 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

• Sign RPMs with SHA256 for FIPs compatbility. (@mattdurham)
• Fix issue where corrupt WAL segments lead to crash looping. (@tpaschalis)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.35.3

This is release v0.35.3 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

• Fix a bug which prevented the app_agent_receiver integration from processing traces. (@ptodev)

• Re-enable Jaeger remote sampling in static_mode via a different config entry. (@ptodev)

• Fix graceful termination when receiving SIGTERM/CTRL_SHUTDOWN_EVENT signals. (@tpaschalis)

• Clustering: Nodes take part in distributing load only after loading their component graph. (@tpaschalis)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.35.2

This is release v0.35.2 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Bugfixes

• Fix issue where the flow mode UI would show an empty page when navigating to
  an unhealthy prometheus.operator component or a healthy
  prometheus.operator component which discovered no custom resources.
  (@rfratto)

• Fix panic when using oauth2 without specifying tls_config. (@mattdurham)

• Fix issue where series records would never get written to the WAL if a scrape
  was rolled back, resulting in "dropped sample for series that was not
  explicitly dropped via relabelling" log messages. (@rfratto)

• Fix RPM file digests so that installation on FIPS-enabled systems succeeds. (@andrewimeson)

Other changes

• Compile journald support into builds of grafana-agentctl so
  grafana-agentctl test-logs functions as expected when testing tailing the
  systemd journal. (@rfratto)

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode

v0.35.1

This is release v0.35.1 of Grafana Agent.

Upgrading

Read the relevant upgrade guides for specific instructions on upgrading from older versions:

• Static mode upgrade guide
• Static mode Kubernetes operator upgrade guide
• Flow mode upgrade guide

Notable changes:

Bugfixes

• Fix incorrect display of trace IDs in the automatic_logging processor of static mode's traces subsystem.
  Users of the static mode's service_graphs processor are also advised to upgrade,
  although the bug should theoretically not affect them. (@ptodev)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guides for how to install the variants of Grafana Agent:

• Install static mode
• Install the static mode Kubernetes operator
• Install flow mode