Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pyroscope task doc #6133

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/sources/flow/tasks/collect-prometheus-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,8 @@ prometheus.remote_write "default" {
}
```

For more information on configuring Kubernetes service delivery and collecting metrics, refer to [discovery.kubernetes][] and [prometheus.scrape][].
For more inf- [ ] Tests updated
- [ ] Config converters updated](https://docs.docker.com/engine/install/)ormation on configuring Kubernetes service delivery and collecting metrics, refer to [discovery.kubernetes][] and [prometheus.scrape][].

## Collect metrics from custom targets

Expand Down
132 changes: 132 additions & 0 deletions docs/sources/flow/tasks/send-profiles-to-pyroscope/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
---
aliases:
- /docs/grafana-cloud/agent/flow/tasks/send-profiles-to-pyroscope/
canonical: https://grafana.com/docs/agent/latest/flow/send-profiles-to-pyroscope/
description: Send profiles to Pyroscope
title: Send profiles to Pyroscope
weight: 120
---

# Send profiles to Pyroscope

Learn how to configure {{< param "PRODUCT_NAME" >}} to collect profiles and forward them to a [Pyroscope Server][].

This topic describes how to:

* Setup [Pyroscope Server][].
* Setup {{< param "PRODUCT_NAME" >}} to collect profiles and send them to [Pyroscope Server][].

## Components used in this topic

* [pyroscope.write][]
* [pyroscope.scrape][]initial

## Before you begin

* Install [docker][].

## Steps
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a tutorial to me. See https://github.com/grafana/agent/blob/main/docs/developer/writing-docs.md

A tutorial:

Provides procedures that users can safely reproduce and learn from. Answers the question: “Can you teach me to …?”

While task:

Tasks are production-ready and contain best practices and recommendations. They are quite detailed, with Reference pages being the only type of documentation that has more detail.

I believe we want a task doc, which will help users configure sending to Pyroscope in production, instead of on a local machine. I know that config is roughly the same, but there's a distinction in our docs now between tasks and tutorials that I'd like to maintain.

cc @clayton-cornell to chime in, if we need a second opinion ;)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thampiotr @mattdurham Yes, I was thinking the same thing. That's one of the reasons I want to re-review this PR once some of the questions I raised are cleared up.

I agree, it's more tutorial than task. For the tasks we need to define what the purpose or goal is, and it should generally be a single thing... a single goal such as (only) configure an existing setup to send Pyroscope in production. We make quite a few assumptions such as... Agent is already installed... and we focus on one thing... the configuration needs.


1. Setup Pyroscope

The easiest path to setting up Pyroscope is using `docker run -it -p 4040:4040 grafana/pyroscope`. This will spin up a Docker instance running Pyroscope on port `4040`. The server will autoscrape itself, and after a few minutes, data will appear. If you immediately access the server, it may error with `No applications available` until it has scraped itself.

![Initial Pyroscope server screen](/media/oss/agent/initial-pyro.png)

2. Install the [latest][] version of {{< param "PRODUCT_NAME" >}} for your operating system.

3. Add the below configuration to `agent.river` file in the same directory as the file downloaded above.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. Add the below configuration to `agent.river` file in the same directory as the file downloaded above.
3. Add the following configuration to the `agent.river` file in the same directory as the file downloaded above.

I'm confused by this... you download the latest Agent... then save the River config file to the same directory as the downloaded Agent? Why? I would expect to add the config file to the existing config or to the default config location. Am I misunderstanding what's oing on here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no default config location, run PATH must always be specified. With the services we do encode that into the service definition. There is no requirement to be in the same directory, but if we assume it is then it makes the commands more copy pastable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. The wording here still doesn't make sense to me. maybe it'll resolve itself if this is reworked as a task. If we keep or expand it as a tutorial we'd need to fix this up and use the path either as a replaceable variable or explicitly name a location to use for the tutorial.


```river
pyroscope.scrape "agent" {
targets = [{"__address__" = "localhost:12345", "service_name" = "agent"}]
forward_to = [pyroscope.write.local.receiver]

profiling_config {
profile.process_cpu {
enabled = true
}

profile.godeltaprof_memory {
enabled = true
}
}
}

pyroscope.write "local" {
endpoint {
url = "http://localhost:4040"
}
}

```

This configuration will scrape the `localhost:12345/-/pprof` endpoint for CPU and memory data every 60 seconds and send those profiles to the Pyroscope server. The `localhost:12345` endpoint is the default host and port for {{< param "PRODUCT_NAME" >}}.

4. Run {{< param "PRODUCT_NAME" >}} with `AGENT_MODE=flow ./grafana-agent-linux-amd64 run ./agent.river`. The exact executable name will change depending on the platform. Wait 2 minutes, this will give time for startup and a scrape to occur.

5. Open `http://localhost:4040` in a web browser.

6. Select `agent` from the dropdown. This name is derived from `service_name` specified in `agent.river`.

![Select agent from dropdown](/media/oss/agent/select-pyro.png)

7. Select any CPU or metric you want to view.

![Agent CPU](/media/oss/agent/normal-pyro.png)

## Using Pyroscope with Grafana Cloud

1. Login to your account at [Grafana Cloud][]- [ ] Tests updated
- [ ] Config converters updated](https://docs.docker.com/engine/install/)
2. Go to your stack and select Details.
3. Select Details under the Pyroscope logo.
4. Generate an API key.
5. Update the `agent.river` file to look like the following configuration. Fill in `URL`,`USERNAME`, and `PASSWORD` with the information from the Pyroscope details page.

```river
pyroscope.scrape "agent" {
t- [ ] Tests updated
- [ ] Config converters updated](https://docs.docker.com/engine/install/)argets = [{"__address__" = "localhost:12345", "service_name" = "agent"}]
forward_to = [pyroscope.write.local.receiver]

profiling_config {
profile.process_cpu {
enabled = true
}

profile.godeltaprof_memory {
enabled = true
}
}
}

pyroscope.write "local" {
endpoint {
url = URL
basic_auth {
username = USERNAME
password = PASSWORD
}
}
}
```



## Additional links

* [Set up Go profiling in pull mode][]

[latest]: https://github.com/grafana/agent/releases/latest
[Set up Go profiling in pull mode]: https://grafana.com/docs/pyroscope/v1.2.x/configure-client/grafana-agent/go_pull/
[Pyroscope Server]: https://github.com/grafana/pyroscope#-quick-start-run-pyroscope-locally
[Grafana Cloud]: https://grafana.com/
[docker]: https://docs.docker.com/engine/install/

{{% docs/reference %}}
[pyroscope.write]: "/docs/agent/ -> /docs/agent/<AGENT_VERSION>/flow/reference/components/pyroscope.write.md"
[pyroscope.write]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/send-data/agent/flow/reference/components/pyroscope.write.md"
[pyroscope.scrape]: "/docs/agent/ -> /docs/agent/<AGENT_VERSION>/flow/reference/components/pyroscope.scrape.md"
[pyroscope.scrape]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/send-data/agent/flow/reference/components/pyroscope.scrape.md"
{{% /docs/reference %}}
Loading