From 8ab541e922d22faf7809851ad5fac3742d3e06dc Mon Sep 17 00:00:00 2001 From: Paulin Todev Date: Fri, 24 Nov 2023 13:50:00 +0000 Subject: [PATCH] Clarify some aspects of pyroscope.scrape --- .../reference/components/pyroscope.scrape.md | 366 ++++++++++++------ 1 file changed, 257 insertions(+), 109 deletions(-) diff --git a/docs/sources/flow/reference/components/pyroscope.scrape.md b/docs/sources/flow/reference/components/pyroscope.scrape.md index 1d7e514b6732..bbf719c94bc5 100644 --- a/docs/sources/flow/reference/components/pyroscope.scrape.md +++ b/docs/sources/flow/reference/components/pyroscope.scrape.md @@ -15,12 +15,35 @@ title: pyroscope.scrape {{< docs/shared lookup="flow/stability/beta.md" source="agent" version="" >}} -`pyroscope.scrape` configures a [pprof] scraping job for a given set of -`targets`. The scraped performance profiles are forwarded to the list of receivers passed in -`forward_to`. +`pyroscope.scrape` collects [pprof] performance profiles for a given set of HTTP `targets`. + +`pyroscope.scrape` borrows the scraping behavior of `prometheus.scrape`. +Similarly to how Prometheus scrapes metrics via HTTP, `pyroscope.scrape` collects profiles via HTTP requests. + +Unlike how Prometheus usually only scrapes one `/metrics` endpoint per target, +`pyroscope.scrape` may need to scrape multiple endpoints for the same target. +This is because different types of profiles are scraped on different endpoints. +For example, "mutex" profiles may be scraped on a `/debug/pprof/delta_mutex` HTTP endpoint, whereas +memory consumption may be scraped on a `/debug/pprof/allocs` HTTP endpoint. + +The profile paths, protocol scheme, scrape interval, scrape timeout, +query parameters, as well as any other settings can be configured within `pyroscope.scrape`. + +The `pyroscope.scrape` component regards a scrape as successful if it +responded with an HTTP `200 OK` status code and returned the body of a valid [pprof] profile. + +If a scrape request fails, the [debug UI][] for `pyroscope.scrape` will show: +* Detailed information about the failure. +* The last successful scrape. +* The labels last used for scraping. + +The scraped performance profiles can be forwarded to components such as +`pyroscope.write` via the `forward_to` argument. Multiple `pyroscope.scrape` components can be specified by giving them different labels. +[debug UI]: {{< relref "../../monitoring/debugging/" >}} + ## Usage ```river @@ -32,45 +55,104 @@ pyroscope.scrape "LABEL" { ## Arguments -The component configures and starts a new scrape job to scrape all of the -input targets. Multiple scrape jobs can be spawned for a single input target +`pyroscope.scrape` starts a new scrape job to scrape all of the input targets. +Multiple scrape jobs can be started for a single input target when scraping multiple profile types. The list of arguments that can be used to configure the block is presented below. -The scrape job name defaults to the component's unique identifier. - -Any omitted fields take on their default values. If conflicting -attributes are being passed (e.g., defining both a BearerToken and -BearerTokenFile or configuring both Basic Authorization and OAuth2 at the same -time), the component reports an error. +Any omitted arguments take on their default values. If conflicting +arguments are being passed (for example, configuring both `bearer_token` +and `bearer_token_file`), then `pyroscope.scrape` will fail to start and will report an error. The following arguments are supported: -Name | Type | Description | Default | Required ----- | ---- | ----------- | ------- | -------- -`targets` | `list(map(string))` | List of targets to scrape. | | yes -`forward_to` | `list(ProfilesReceiver)` | List of receivers to send scraped profiles to. | | yes -`job_name` | `string` | The job name to override the job label with. | component name | no -`params` | `map(list(string))` | A set of query parameters with which the target is scraped. | | no -`scrape_interval` | `duration` | How frequently to scrape the targets of this scrape config. | `"15s"` | no -`scrape_timeout` | `duration` | The timeout for scraping targets of this config. | `"15s"` | no -`scheme` | `string` | The URL scheme with which to fetch metrics from targets. | | no -`bearer_token` | `secret` | Bearer token to authenticate with. | | no -`bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no -`proxy_url` | `string` | HTTP proxy to proxy requests through. | | no -`follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no -`enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no - - At most one of the following can be provided: +Name | Type | Description | Default | Required +------------------- | ------------------------ | ------------------------------------------------------------ | -------------- | -------- +`targets` | `list(map(string))` | List of targets to scrape. | | yes +`forward_to` | `list(ProfilesReceiver)` | List of receivers to send scraped profiles to. | | yes +`job_name` | `string` | The job name to override the job label with. | component name | no +`params` | `map(list(string))` | A set of query parameters with which the target is scraped. | | no +`scrape_interval` | `duration` | How frequently to scrape the targets of this scrape config. | `"15s"` | no +`scrape_timeout` | `duration` | The timeout for scraping targets of this config. | `"18s"` | no +`scheme` | `string` | The URL scheme with which to fetch metrics from targets. | `"http"` | no +`bearer_token` | `secret` | Bearer token to authenticate with. | | no +`bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no +`proxy_url` | `string` | HTTP proxy to proxy requests through. | | no +`follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no +`enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no + +At most one of the following authentication mechanisms can be provided: - [`bearer_token` argument](#arguments). - [`bearer_token_file` argument](#arguments). - [`basic_auth` block][basic_auth]. - [`authorization` block][authorization]. - [`oauth2` block][oauth2]. - [arguments]: #arguments +[arguments]: #arguments + +#### `job_name` argument + +`job_name` defaults to the component's unique identifier. + +For example, the `job_name` of `pyroscope.scrape "local" { ... }` will be `"pyroscope.scrape/local"`. + +#### `targets` argument + +The list of `targets` can be provided [statically][example_static_targets], [dynamically][example_dynamic_targets], +or a [combination of both][example_static_and_dynamic_targets]. + +The special `__address__` label _must always_ be present and corresponds to the +`:` that is used for the scrape request. + +Labels starting with a double underscore (`__`) are treated as _internal_, and are removed prior to scraping. + +The special label `service_name` is required and must always be present. +If it is not specified, `pyroscope.scrape` will attempt to infer it from +either of the following sources, in this order: +1. `__meta_kubernetes_pod_annotation_pyroscope_io_service_name` which is a `pyroscope.io/service_name` pod annotation. +2. `__meta_kubernetes_namespace` and `__meta_kubernetes_pod_container_name` +3. `__meta_docker_container_name` + +If `service_name` is not specified and could not be inferred, then it is set to `unspecified`. + +The following labels are automatically injected to the scraped profiles +so that they can be linked to a scrape target: + +| Label | Description | +|------------------|----------------------------------------------------------------- | +| `"job"` | The `job_name` that the target belongs to. | +| `"instance"` | The `__address__` or `:` of the scrape target's URL. | +| `"service_name"` | The inferred Pyroscope service name. | + +#### `scrape_interval` argument + +If `scrape_interval` is short: +* Advantages: + * Less profiles may be lost if the application being scraped crashes. +* Disadvantages: + * Greater network load during scrapes and remote writes. + +If `scrape_interval` is long: +* Advantages: + * More profiles may be lost if the application being scraped crashes. +* Disadvantages: + * If the [delta argument][] is set to `true`, the batch size of + each remote write to Pyroscope may be bigger. + The Pyroscope database may need to be tuned with higher limits. + * If the [delta argument][] is set to `true`, there is a larger risk of + reaching the HTTP server timeouts of the application being scraped. + +For example, consider this situation: +* `pyroscope.scrape` is configured with a `scrape_interval` of `"60s"`. +* The application being scraped is running an HTTP server with a timeout of 30 seconds. +* Any scrape HTTP requests where the [delta argument][] is set to `true` will fail, + because they will attempt to run for 59 seconds. + +#### `scrape_timeout` argument + +`scrape_timeout` should be larger than `scrape_interval`. ## Blocks @@ -92,7 +174,7 @@ The following blocks are supported inside the definition of `pyroscope.scrape`: | profiling_config > profile.fgprof | [profile.fgprof][] | Collect [fgprof][] profiles. | no | | profiling_config > profile.godeltaprof_memory | [profile.godeltaprof_memory][] | Collect [godeltaprof][] memory profiles. | no | | profiling_config > profile.godeltaprof_mutex | [profile.godeltaprof_mutex][] | Collect [godeltaprof][] mutex profiles. | no | -| profiling_config > profile.godeltaprof_block | [profile.godeltaprof_block][] | Collect [godeltaprof][] block profiles. | no | +| profiling_config > profile.godeltaprof_block | [profile.godeltaprof_block][] | Collect [godeltaprof][] block profiles. | no | | profiling_config > profile.custom | [profile.custom][] | Collect custom profiles. | no | | clustering | [clustering][] | Configure the component for when the Agent is running in clustered mode. | no | @@ -100,27 +182,33 @@ The `>` symbol indicates deeper levels of nesting. For example, `oauth2 > tls_config` refers to a `tls_config` block defined inside an `oauth2` block. +Any omitted blocks take on their default values. For example, +if `profile.mutex` is not specified in the config, +the defaults documented in [profile.mutex][] will be used. + [basic_auth]: #basic_auth-block [authorization]: #authorization-block [oauth2]: #oauth2-block [tls_config]: #tls_config-block [profiling_config]: #profiling_config-block -[profile.memory]: #profile.memory-block -[profile.block]: #profile.block-block -[profile.goroutine]: #profile.goroutine-block -[profile.mutex]: #profile.mutex-block -[profile.process_cpu]: #profile.process_cpu-block -[profile.fgprof]: #profile.fgprof-block -[profile.godeltaprof_memory]: #profile.godeltaprof_memory-block -[profile.godeltaprof_mutex]: #profile.godeltaprof_mutex-block -[profile.godeltaprof_block]: #profile.godeltaprof_block-block -[profile.custom]: #profile.custom-block +[profile.memory]: #profilememory-block +[profile.block]: #profileblock-block +[profile.goroutine]: #profilegoroutine-block +[profile.mutex]: #profilemutex-block +[profile.process_cpu]: #profileprocess_cpu-block +[profile.fgprof]: #profilefgprof-block +[profile.godeltaprof_memory]: #profilegodeltaprof_memory-block +[profile.godeltaprof_mutex]: #profilegodeltaprof_mutex-block +[profile.godeltaprof_block]: #profilegodeltaprof_block-block +[profile.custom]: #profilecustom-block [pprof]: https://github.com/google/pprof/blob/main/doc/README.md [clustering]: #clustering-beta [fgprof]: https://github.com/felixge/fgprof [godeltaprof]: https://github.com/grafana/godeltaprof +[delta argument]: #delta-argument + ### basic_auth block {{< docs/shared lookup="flow/reference/components/basic-auth-block.md" source="agent" version="" >}} @@ -142,7 +230,7 @@ an `oauth2` block. The `profiling_config` block configures the profiling settings when scraping targets. -The block contains the following attributes: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -152,7 +240,7 @@ Name | Type | Description | Default | Required The `profile.memory` block collects profiles on memory consumption. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -160,14 +248,13 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/pprof/allocs"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `false` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.block block The `profile.block` block collects profiles on process blocking. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -175,14 +262,13 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/pprof/block"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `false` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.goroutine block The `profile.goroutine` block collects profiles on the number of goroutines. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -190,14 +276,13 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/pprof/goroutine"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `false` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.mutex block The `profile.mutex` block collects profiles on mutexes. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -205,15 +290,14 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/pprof/mutex"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `false` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.process_cpu block The `profile.process_cpu` block collects profiles on CPU consumption for the process. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -221,14 +305,13 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/pprof/profile"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `true` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.fgprof block The `profile.fgprof` block collects profiles from an [fgprof][] endpoint. -It accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -236,14 +319,13 @@ Name | Type | Description | Default | Required `path` | `string` | The path to the profile type on the target. | `"/debug/fgprof"` | no `delta` | `boolean` | Whether to scrape the profile as a delta. | `true` | no -When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +For more information about the `delta` argument, see the [delta argument][] section. ### profile.godeltaprof_memory block The `profile.godeltaprof_memory` block collects profiles from [godeltaprof][] memory endpoint. The delta is computed on the target. -It accepts the following arguments: +The following arguments are supported: | Name | Type | Description | Default | Required | |-----------|-----------|---------------------------------------------|-----------------------------|----------| @@ -254,7 +336,7 @@ It accepts the following arguments: The `profile.godeltaprof_mutex` block collects profiles from [godeltaprof][] mutex endpoint. The delta is computed on the target. -It accepts the following arguments: +The following arguments are supported: | Name | Type | Description | Default | Required | |-----------|-----------|---------------------------------------------|------------------------------|----------| @@ -265,7 +347,7 @@ It accepts the following arguments: The `profile.godeltaprof_block` block collects profiles from [godeltaprof][] block endpoint. The delta is computed on the target. -It accepts the following arguments: +The following arguments are supported: | Name | Type | Description | Default | Required | |-----------|-----------|---------------------------------------------|------------------------------|----------| @@ -288,7 +370,7 @@ profile.custom "PROFILE_TYPE" { Multiple `profile.custom` blocks can be specified. Labels assigned to `profile.custom` blocks must be unique across the component. -The `profile.custom` block accepts the following arguments: +The following arguments are supported: Name | Type | Description | Default | Required ---- | ---- | ----------- | ------- | -------- @@ -297,7 +379,7 @@ Name | Type | Description | Default | Required `delta` | `boolean` | Whether to scrape the profile as a delta. | `false` | no When the `delta` argument is `true`, a `seconds` query parameter is -automatically added to requests. +automatically added to requests. The `seconds` used will be equal to `scrape_interval - 1`. ### clustering (beta) @@ -322,6 +404,19 @@ If the agent is _not_ running in clustered mode, this block is a no-op. [using clustering]: {{< relref "../../concepts/clustering.md" >}} +## Common configuration + +### `delta` argument + +When the `delta` argument is `false`, the [pprof][] HTTP query will be instantaneous. + +When the `delta` argument is `true`: +* The [pprof][] HTTP query will run for a certain amount of time. +* A `seconds` parameter is automatically added to the HTTP request. +* The `seconds` used will be equal to `scrape_interval - 1`. + For example, if `scrape_interval` is `"15s"`, `seconds` will be 14 seconds. + If the HTTP endpoint is `/debug/pprof/profile`, then the HTTP query will become `/debug/pprof/profile?seconds=14` + ## Exported fields `pyroscope.scrape` does not export any fields that can be referenced by other @@ -341,67 +436,115 @@ scrape job on the component's debug endpoint. * `pyroscope_fanout_latency` (histogram): Write latency for sending to direct and indirect components. -## Scraping behavior +## Examples -The `pyroscope.scrape` component borrows the scraping behavior of Prometheus. -Prometheus, and by extension, this component, uses a pull model for scraping -profiles from a given set of _targets_. -Each scrape target is defined as a set of key-value pairs called _labels_. +[example_static_targets]: #default-endpoints-of-static-targets -The set of targets can either be _static_, or dynamically provided periodically -by a service discovery component such as `discovery.kubernetes`. The special -label `__address__` _must always_ be present and corresponds to the -`:` that is used for the scrape request. +### Default endpoints of static targets + +The following example sets up a scrape job of a statically configured +list of targets - the Agent itself and Pyroscope. +The scraped profiles are sent to `pyroscope.write` which remote writes them to a Pyroscope database. + +```river +pyroscope.scrape "local" { + targets = [ + {"__address__" = "localhost:4100", "service_name"="pyroscope"}, + {"__address__" = "localhost:12345", "service_name"="agent"}, + ] + + forward_to = [pyroscope.write.local.receiver] +} -The special label `service_name` is required and must always be present. If it's not specified, it is -attempted to be inferred from multiple sources: -- `__meta_kubernetes_pod_annotation_pyroscope_io_service_name` which is a `pyroscope.io/service_name` pod annotation. -- `__meta_kubernetes_namespace` and `__meta_kubernetes_pod_container_name` -- `__meta_docker_container_name` +pyroscope.write "local" { + endpoint { + url = "http://pyroscope:4100" + } +} +``` -If `service_name` is not specified and could not be inferred it is set to `unspecified`. +These endpoints will be scraped every 15 seconds: -By default, the scrape job tries to scrape all available targets' `/debug/pprof` -endpoints using HTTP, with a scrape interval of 15 seconds and scrape timeout of -15 seconds. The profile paths, protocol scheme, scrape interval and timeout, -query parameters, as well as any other settings can be configured using the -component's arguments. +``` +http://localhost:4100/debug/pprof/allocs +http://localhost:4100/debug/pprof/block +http://localhost:4100/debug/pprof/goroutine +http://localhost:4100/debug/pprof/mutex +http://localhost:4100/debug/pprof/profile?seconds=14 -The scrape job expects profiles exposed by the endpoint to follow the -[pprof] protobuf format. All profiles are then propagated -to each receiver listed in the component's `forward_to` argument. +http://localhost:12345/debug/pprof/allocs +http://localhost:12345/debug/pprof/block +http://localhost:12345/debug/pprof/goroutine +http://localhost:12345/debug/pprof/mutex +http://localhost:12345/debug/pprof/profile?seconds=14 +``` -Labels coming from targets, that start with a double underscore `__` are -treated as _internal_, and are removed prior to scraping. +Note that `seconds=14` is added to the `/debug/pprof/profile` endpoint, because: +* The `delta` argument of the `profile.process_cpu` block is `true` by default. +* `scrape_interval` is `"15s"` by default. -The `pyroscope.scrape` component regards a scrape as successful if it -responded with an HTTP `200 OK` status code and returned a body of valid [pprof] profile. +Also note that the `/debug/fgprof` endpoint will not be scraped, because +the `enabled` argument of the `profile.fgprof` block is `false` by default. -If the scrape request fails, the component's debug UI section contains more -detailed information about the failure, the last successful scrape, as well as -the labels last used for scraping. +[example_dynamic_targets]: #default-endpoints-of-dynamic-targets -The following labels are automatically injected to the scraped profiles and -can help pin down a scrape target. +### Default endpoints of dynamic targets -| Label | Description | -|--------------|--------------------------------------------------------------------------------------------------| -| job | The configured job name that the target belongs to. Defaults to the fully formed component name. | -| instance | The `__address__` or `:` of the scrape target's URL. | -| service_name | The inferred pyroscope service name | +```river +discovery.http "dynamic_targets" { + url = "https://example.com/scrape_targets" + refresh_interval = "15s" +} -## Example +pyroscope.scrape "local" { + targets = [discovery.http.dynamic_targets.targets] -The following example sets up the scrape job with certain attributes (profiling config, targets) and lets it scrape two local applications (the Agent itself and Pyroscope). -The exposed profiles are sent over to the provided list of receivers, as defined by other components. + forward_to = [pyroscope.write.local.receiver] +} + +pyroscope.write "local" { + endpoint { + url = "http://pyroscope:4100" + } +} +``` + +[example_static_and_dynamic_targets]: #default-endpoints-of-static-and-dynamic-targets + +### Default endpoints of static and dynamic targets ```river +discovery.http "dynamic_targets" { + url = "https://example.com/scrape_targets" + refresh_interval = "15s" +} + pyroscope.scrape "local" { - targets = [ + targets = [ {"__address__" = "localhost:4100", "service_name"="pyroscope"}, {"__address__" = "localhost:12345", "service_name"="agent"}, + discovery.http.dynamic_targets.targets, ] + forward_to = [pyroscope.write.local.receiver] +} + +pyroscope.write "local" { + endpoint { + url = "http://pyroscope:4100" + } +} +``` + + +### Enabling and disabling profiles + +```river +pyroscope.scrape "local" { + targets = [ + {"__address__" = "localhost:12345", "service_name"="agent"}, + ] + profiling_config { profile.fgprof { enabled = true @@ -413,18 +556,23 @@ pyroscope.scrape "local" { enabled = false } } + + forward_to = [pyroscope.write.local.receiver] } ``` -Here are the endpoints that are being scraped every 15 seconds: +These endpoints will be scraped every 15 seconds: ``` -http://localhost:4100/debug/pprof/allocs -http://localhost:4100/debug/pprof/goroutine -http://localhost:4100/debug/pprof/profile?seconds=14 -http://localhost:4100/debug/fgprof?seconds=14 http://localhost:12345/debug/pprof/allocs http://localhost:12345/debug/pprof/goroutine http://localhost:12345/debug/pprof/profile?seconds=14 http://localhost:12345/debug/fgprof?seconds=14 ``` + +These endpoints will **NOT** be scraped because they are explicitly disabled: + +``` +http://localhost:12345/debug/pprof/block +http://localhost:12345/debug/pprof/mutex +``` \ No newline at end of file