Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs to clarify what various components are used for #6182

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions docs/sources/flow/reference/components/_index.md
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't mentioned some components yet:

I'm worried that this page is starting to get really long. Maybe we could make it a totally separate page?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would start with the single page in draft and split only if needed. As it is, each section is fairly small

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But... as I read it we are getting really deep with the heading levels. I would suggest we break this off the index and just have it as its own standalone page. That might help reduce the empty heading overload that the current draft has (we should avoid having a heading immediately followed by another heading with no text in between the headings).

Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,96 @@ weight: 300

This section contains reference documentation for all recognized [components][].

## Which component do I need?

### Receiving telemetry

##### Logs

* In the Loki format: `loki.source` components such as `loki.source.file`.
* If you need to convert from the Loki format to OTLP, you can use `otelcol.receiver.loki`.
* In the OTLP format: some `otelcol.receiver` components, such as `otelcol.receiver.otlp`.

##### Metrics

`prometheus.scrape` scrapes HTTP endpoints using the Prometheus exposition format.

`prometheus.receive_http` accepts metrics in the Prometheus format on a HTTP server.
A separate process such as {{< param "PRODUCT_ROOT_NAME" >}}, Prometheus, or OpenTelemetry Collector could remote write ("push") metrics on this server.

##### Traces

* In the OTLP format: some `otelcol.receiver` components such as `otelcol.receiver.otlp`.

##### Profiles

`pyroscope.scrape` scrapes profiles from HTTP endpoints using the Pprof format.

### Processing telemetry

#### Logs

All `loki.source` components can receive logs, in the Loki format. If you need to convert from the Loki format to OTLP, you can use `otelcol.receiver.loki`.

##### Metrics

* In the Prometheus format: `prometheus.relabel`.
* In the OTLP format: `otelcol.processor` components such as `otelcol.processor.transform`.

##### Traces

* In the OTLP format: `otelcol.processor` components such as `otelcol.processor.transform`.

##### Profiles

At this time, profiles can only be received and sent by {{< param "PRODUCT_ROOT_NAME" >}}.

### Sending telemetry

##### Logs

All `loki.source` components can receive logs, in the Loki format. If you need to convert from the Loki format to OTLP, you can use `otelcol.receiver.loki`.

Some `otelcol.receiver` components such as `otelcol.receiver.otlp` can also receive logs, in the OTLP format.

##### Metrics

In the Prometheus format: `prometheus.remote_write`.

In the OTLP format: `otelcol.exporter` components such as `otelcol.exporter.otlp`.

##### Traces

* In the OTLP format: `otelcol.exporter` components such as `otelcol.exporter.otlp`.

##### Profiles

`pyroscope.write` sends profiles using Pyroscope's Push API.

### Converting one telemetry type to another

`loki.process` can create metrics out of logs using the `stage.metrics` block.

`otelcol.connector` components generally output a different type of telemetry from the one that they input.
For example, `otelcol.connector.spanmetrics` creates RED OTLP metrics from OTLP traces.

Converting metrics from the native Prometheus format to OTLP can be done using `otelcol.receiver.prometheus`.
To convert OTLP metrics to Prometheus, use `otelcol.exporter.prometheus`.

Converting logs from the native Loki format to OTLP can be done using `otelcol.receiver.loki`.
To convert OTLP metrics to Prometheus, use `otelcol.exporter.loki`.

### Configuring {{< param "PRODUCT_ROOT_NAME" >}} dynamically

The Agent can receive part of all of its configuration dynamically, via `module` components such as `module.file`.
`module.git` pulls AGent configuration from a Git repository, whereas `module.http` pulls it from an HTTP endpoint.

### Other components

`mimir.rules.kubernetes` discovers `PrometheusRule` Kubernetes resources and loads them into a Mimir instance.

## All components

{{< section >}}

[components]: {{< relref "../../concepts/components.md" >}}
Loading