Beyla integration test

.github/workflows/integration-tests.yml

@@ -1,15 +1,15 @@
 name: Integration Tests
 on:
-  # Run tests on main just so the module and build cache can be saved and used
-  # in PRs. This speeds up the time it takes to test PRs dramatically.
-  # (More information on https://docs.github.com/en/[email protected]/actions/using-workflows/caching-dependencies-to-speed-up-workflows)
   push:
     branches:
       - main
   pull_request:
 jobs:
   run_tests:
     runs-on: ubuntu-latest
+    container:
+      image: ubuntu:latest
+      options: --privileged
     steps:
     - name: Checkout code
       uses: actions/checkout@v4
@@ -20,4 +20,4 @@ jobs:
     - name: Set OTEL Exporter Endpoint
       run: echo "OTEL_EXPORTER_ENDPOINT=172.17.0.1:4318" >> $GITHUB_ENV
     - name: Run tests
-      run: make integration-test
+      run: cd internal/cmd/integration-tests && go run .

CHANGELOG.md

@@ -41,6 +41,8 @@ Main (unreleased)
 
 - A new `loki.rules.kubernetes` component that discovers `PrometheusRule` Kubernetes resources and loads them into a Loki Ruler instance. (@EStork09)
 
+- Add `beyla.ebpf` component to automatically instrument services with eBPF. (@marctc)
+
 ### Bugfixes
 
 - Fix an issue where JSON string array elements were not parsed correctly in `loki.source.cloudflare`. (@thampiotr)

CODEOWNERS

@@ -18,4 +18,5 @@
 /docs/sources/ @clayton-cornell
 
 # Components:
-/component/pyroscope/ @grafana/grafana-agent-profiling-maintainers
+/internal/component/pyroscope/ @grafana/grafana-agent-profiling-maintainers
+/internal/component/beyla/     @marctc

docs/sources/flow/reference/compatibility/_index.md

@@ -44,6 +44,10 @@ The following components, grouped by namespace, _export_ Targets.
 
 <!-- START GENERATED SECTION: EXPORTERS OF Targets -->
 
+{{< collapse title="beyla" >}}
+- [beyla.ebpf](../components/beyla.ebpf)
+{{< /collapse >}}
+
 {{< collapse title="discovery" >}}
 - [discovery.azure](../components/discovery.azure)
 - [discovery.consul](../components/discovery.consul)
@@ -315,6 +319,10 @@ The following components, grouped by namespace, _consume_ OpenTelemetry `otelcol
 
 <!-- START GENERATED SECTION: CONSUMERS OF OpenTelemetry `otelcol.Consumer` -->
 
+{{< collapse title="beyla" >}}
+- [beyla.ebpf](../components/beyla.ebpf)
+{{< /collapse >}}
+
 {{< collapse title="faro" >}}
 - [faro.receiver](../components/faro.receiver)
 {{< /collapse >}}

docs/sources/flow/reference/components/beyla.ebpf.md

@@ -0,0 +1,270 @@
+---
+canonical: https://grafana.com/docs/agent/latest/flow/reference/components/beyla.ebpf/
+description: Learn about beyla.ebpf
+title: beyla.ebpf
+---
+
+# beyla.ebpf
+
+The `beyla.ebpf` component is used a wrapper of [Grafana Beyla][] which uses [eBPF][] to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. The component can be configured to collect telemetry data from a specific port or executable path, and other criteria from Kubernetes metadata. The component exposes metrics which can be collected by a Prometheus scrape component, and traces which can be forwared to a OTEL exporter component.
+
+{{< admonition type="note" >}}
+To run this component, {{< param "PRODUCT_NAME" >}} requires administrative (`sudo`) privileges, or at least it needs to be granted the `CAP_SYS_ADMIN` and `CAP_SYS_PTRACE` capability. In Kubernetes environments, app armour must be disabled for the Deployment or DaemonSet running {{< param "PRODUCT_NAME" >}}.
+{{< /admonition >}}
+
+[Grafana Beyla]: https://github.com/grafana/beyla
+[eBPF]: https://ebpf.io/
+
+## Usage
+
+```river
+beyla.ebpf "<LABEL>" {
+
+}
+```
+
+## Arguments
+
+`beyla.ebpf` supports the following arguments:
+
+Name | Type | Description                                               | Default | Required
+---- | ---- |-----------------------------------------------------------| ------- | --------
+`open_port` | `string` | The port of the running service for Beyla automatically instrumented with eBPF. | | no
+`excutable_name` | `string` | The name of the executable to match for Beyla automatically instrumented with eBPF. | | no
+
+
+`open_port` accepts a comma-separated list of ports (for example, `80,443`), and port ranges (for example, `8000-8999`).
+If the executable matches only one of the ports in the list, it is considered to match the selection criteria.
+
+`excutable_name` accepts a regular expression to be matched against the full executable command line, including the directory where the executable resides on the file system.
+
+## Blocks
+
+The following blocks are supported inside the definition of `beyla.ebpf`:
+
+Hierarchy | Block | Description | Required
+--------- | ----- | ----------- | --------
+routes | [routes][] | Configures the routes to match HTTP paths into user-provided HTTP routes. | no
+attributes | [attributes][] | Configures the Beyla attributes for the component. | no
+attributes > kubernetes | [kubernetes][] | Configures decorationg of the metrics and traces with Kubernetes metadata of the instrumented Pods. | no
+discovery | [discovery][] | Configures the discovery for instrumentable processes matching a given criteria. | no
+discovery > services | [services][] | Configures the discovery for the component. | no
+output | [output][] | Configures where to send received telemetry data. | yes
+
+The `>` symbol indicates deeper levels of nesting. For example,
+`attributes > kubernetes` refers to a `kubernetes` block defined inside an
+`attributes` block.
+
+[routes]: #routes-block
+[attributes]: #attributes-block
+[kubernetes]: #kubernetes-block
+[discovery]: #discovery-block
+[services]: #services-block
+[output]: #output-block
+
+### attributes block
+
+Allows configuring how some attributes for metrics and traces are decorated.
+
+Contains the following blocks:
+[kubernetes]: #kubernetes-block
+
+#### kubernetes block
+
+Name | Type | Description                                               | Default | Required
+---- | ---- |-----------------------------------------------------------| ------- | --------
+`enable` | `string` | Enable the Kubernetes metadata decoration. | `false` | no
+
+If set to `true`, Beyla will decorate the metrics and traces with Kubernetes metadata. The following labels will be added:
+
+- `k8s.namespace.name`
+- `k8s.deployment.name`
+- `k8s.statefulset.name`
+- `k8s.replicaset.name`
+- `k8s.daemonset.name`
+- `k8s.node.name`
+- `k8s.pod.name`
+- `k8s.pod.uid`
+- `k8s.pod.start_time`
+
+If set to `false`, the Kubernetes metadata decorator will be disabled.
+
+If set to `autodetect`, Beyla will try to automatically detect if it is running inside Kubernetes, and enable the metadata decoration if that is the case.
+
+
+### routes block
+
+This block is used to configure the routes to match HTTP paths into user-provided HTTP routes.
+
+Name | Type | Description                                               | Default | Required
+---- | ---- |-----------------------------------------------------------| ------- | --------
+`patterns` | `list(string)` | List of provided URL path patterns to set the `http.route` trace/metric property | | no
+`ignore_patterns` | `list(string)` | List of provided URL path patterns to ignore from `http.route` trace/metric property. | | no
+`ignore_mode` | `string` | The mode to use when ignoring patterns. | | no
+`unmatched` | `string` | Specifies what to do when a trace HTTP path does not match any of the `patterns` entries | | no
+
+`patterns` and `ignored_patterns` are a list of patterns which a URL path with specific tags which allow for grouping path segments (or ignored them).
+The matcher tags can be in the `:name` or `{name}` format.
+`ignore_mode` properties are:
+- `all` discards both metrics and traces that match the `ignored_patterns`.
+- `traces` discards only the traces that match the `ignored_patterns`. No metric events are ignored.
+- `metrics` discards only the metrics that match the `ignored_patterns`. No trace events are ignored.
+`unmatched` properties are:
+- `unset` leaves the `http.route` property as unset.
+- `path` copies the `http.route` field property to the path value.
+  - Caution: this option could lead to a cardinality explosion at the ingester side.
+- `wildcard` sets the `http.route` field property to a generic asterisk based `/**` value.
+- `heuristic` automatically derives the `http.route` field property from the path value, based on the following rules:
+  - Any path components that have numbers or characters outside of the ASCII alphabet (or `-` and _), are replaced by an asterisk `*`.
+  - Any alphabetical components that don’t look like words, are replaced by an asterisk `*`.
+
+
+### discovery block
+
+This block is used to configure the discovery for instrumentable processes matching a given criteria.
+
+Contains the following blocks:
+[services]: #services-block
+
+### services block
+
+In some scenarios, Beyla will instrument a big variety of services, for example, as a Kubernetes DaemonSet that instruments all the services in a node.
+This block allows you to filter the services to instrument based on their metadata.
+
+Name | Type | Description                                               | Default | Required
+---- | ---- |-----------------------------------------------------------| ------- | --------
+`name ` | `string` | The name of the service to match. | | no
+`namespace` | `string` | The namespace of the service to match. | | no
+`open_ports` | `string` | The port of the running service for Beyla automatically instrumented with eBPF. | | no
+`path` | `string` | The path of the running service for Beyla automatically instrumented with eBPF. | | no
+
+`name` defines a name for the matching instrumented service. It is used to populate the `service.name` OTEL property and/or the `service_name` Prometheus property in the exported metrics/traces.
+`open_port` accepts a comma-separated list of ports (for example, `80,443`), and port ranges (for example, `8000-8999`). If the executable matches only one of the ports in the list, it is considered to match the selection criteria.
+`path` accepts a regular expression to be matched against the full executable command line, including the directory where the executable resides on the file system.
+
+
+### output block
+
+The `output` block configures a set of components to forward resulting telemetry data to.
+
+The following arguments are supported:
+
+Name      | Type                     | Description                           | Default | Required
+----------|--------------------------|---------------------------------------|---------|---------
+`traces`  | `list(otelcol.Consumer)` | List of consumers to send traces to.  | `[]`    | no
+
+You must specify the `output` block, but all its arguments are optional.
+By default, telemetry data is dropped.
+Configure the `traces` argument accordingly to send traces data to other components.
+
+## Exported fields
+
+The following fields are exported and can be referenced by other components.
+
+Name      | Type                | Description
+----------|---------------------|----------------------------------------------------------
+`targets` | `list(map(string))` | The targets that can be used to collect metrics of instrumented services with eBPF.
+
+For example, the `targets` can either be passed to a `discovery.relabel` component to rewrite the targets' label sets or to a `prometheus.scrape` component that collects the exposed metrics.
+
+The exported targets use the configured [in-memory traffic][] address specified by the [run command][].
+
+[in-memory traffic]: ../../../concepts/component_controller#in-memory-traffic
+[run command]: ../../cli/run/
+
+## Component health
+
+`beyla.ebpf` is only reported as unhealthy if given an invalid
+configuration.
+
+## Debug information
+
+`beyla.ebpf` does not expose any component-specific debug
+information.
+
+## Example
+
+### Metrics
+
+This example uses a [`prometheus.scrape` component][scrape] to collect metrics from `beyla.ebpf` of
+the specified port:
+
+```river
+beyla.ebpf "default" {
+    open_port = <OPEN_PORT>
+}
+
+prometheus.scrape "beyla" {
+  targets = beyla.ebpf.default.targets
+  forward_to = [prometheus.remote_write.mimir.receiver]
+}
+
+prometheus.remote_write "demo" {
+  endpoint {
+    url = <PROMETHEUS_REMOTE_WRITE_URL>
+
+    basic_auth {
+      username = <USERNAME>
+      password = <PASSWORD>
+    }
+  }
+}
+```
+
+Replace the following:
+
+- _`<OPEN_PORT>`_: The port of the running service for Beyla automatically instrumented with eBPF.
+- _`<PROMETHEUS_REMOTE_WRITE_URL>`_: The URL of the Prometheus remote_write-compatible server to send metrics to.
+- _`<USERNAME>`_: The username to use for authentication to the remote_write API.
+- _`<PASSWORD>`_: The password to use for authentication to the remote_write API.
+
+[scrape]: ../prometheus.scrape/
+
+### Traces
+
+This example gets traces from `beyla.ebpf` and forwards them to `otlp`:
+
+```river
+beyla.ebpf "default" {
+    open_port = <OPEN_PORT>
+    output {
+        traces = [otelcol.processor.batch.default.input]
+    }
+}
+
+otelcol.processor.batch "default" {
+    output {
+        traces  = [otelcol.exporter.otlp.default.input]
+    }
+}
+
+otelcol.exporter.otlp "default" {
+    client {
+        endpoint = env("<OTLP_ENDPOINT>")
+    }
+}
+```
+
+Replace the following:
+
+- _`<OPEN_PORT>`_: The port of the running service for Beyla automatically instrumented with eBPF.
+- _`<OTLP_ENDPOINT>`_: The endpoint of the OpenTelemetry Collector to send traces to.
+
+<!-- START GENERATED COMPATIBLE COMPONENTS -->
+
+## Compatible components
+
+`beyla.ebpf` can accept arguments from the following components:
+
+- Components that export [OpenTelemetry `otelcol.Consumer`](../../compatibility/#opentelemetry-otelcolconsumer-exporters)
+
+`beyla.ebpf` has exports that can be consumed by the following components:
+
+- Components that consume [Targets](../../compatibility/#targets-consumers)
+
+{{< admonition type="note" >}}
+Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
+Refer to the linked documentation for more details.
+{{< /admonition >}}
+
+<!-- END GENERATED COMPATIBLE COMPONENTS -->