Skip to content

Commit

Permalink
Merge branch 'main' into tomasmota-patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
tomasmota authored Jul 29, 2024
2 parents 7f466df + 450bbce commit 5fb0980
Show file tree
Hide file tree
Showing 61 changed files with 2,095 additions and 347 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,6 @@ nix/result

# snyk
.dccache

# helm test
production/helm/loki/src/helm-test/helm-test
8 changes: 8 additions & 0 deletions docs/sources/operations/query-acceleration-blooms.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ The underlying blooms are built by the new [Bloom Compactor](#bloom-compactor) c
and served by the new [Bloom Gateway](#bloom-gateway) component.

## Enable Query Acceleration with Blooms
{{< admonition type="warning" >}}
Building and querying bloom filters are by design not supported in single binary deployment.
It can be used with Single Scalable deployment (SSD), but it is recommended to
run bloom components only in fully distributed microservice mode.
The reason is that bloom filters also come with a relatively high cost for both building
and querying the bloom filters that only pays off at large scale deployments.
{{< /admonition >}}

To start building and using blooms you need to:
- Deploy the [Bloom Compactor](#bloom-compactor) component and enable the component in the [Bloom Compactor config][compactor-cfg].
- Deploy the [Bloom Gateway](#bloom-gateway) component (as a [microservice][microservices] or via the [SSD][ssd] Backend target) and enable the component in the [Bloom Gateway config][gateway-cfg].
Expand Down
64 changes: 64 additions & 0 deletions docs/sources/send-data/alloy/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
title: Ingesting logs to Loki using Alloy
menuTitle: Grafana Alloy
description: Configuring Grafana Alloy to send logs to Loki.
weight: 250
---


# Ingesting logs to Loki using Alloy

Grafana Alloy is a versatile observability collector that can ingest logs in various formats and send them to Loki. We recommend Alloy as the primary method for sending logs to Loki, as it provides a more robust and feature-rich solution for building a highly scalable and reliable observability pipeline.

{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Alloy flow diagram" >}}

## Installing Alloy

To get started with Grafana Alloy and send logs to Loki, you need to install and configure Alloy. You can follow the [Alloy documentation](https://grafana.com/docs/alloy/latest/get-started/install/) to install Alloy on your preferred platform.

## Components of Alloy for logs

Alloy pipelines are built using components that perform specific functions. For logs these can be broken down into three categories:

- **Collector:** These components collect/receive logs from various sources. This can be scraping logs from a file, receiving logs over HTTP, gRPC or ingesting logs from a message queue.
- **Transformer:** These components can be used to manipulate logs before they are sent to a writer. This can be used to add additional metadata, filter logs, or batch logs before sending them to a writer.
- **Writer:** These components send logs to the desired destination. Our documentation will focus on sending logs to Loki, but Alloy supports sending logs to various destinations.

### Log components in Alloy

Here is a non-exhaustive list of components that can be used to build a log pipeline in Alloy. For a complete list of components, refer to the [components list](https://grafana.com/docs/alloy/latest/reference/components/).

| Type | Component |
|------------|-----------------------------------------------------------------------------------------------------|
| Collector | [loki.source.api](https://grafana.com/docs/alloy/latest/reference/components/loki.source.api/) |
| Collector | [loki.source.awsfirehose](https://grafana.com/docs/alloy/latest/reference/components/loki.source.awsfirehose/) |
| Collector | [loki.source.azure_event_hubs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.azure_event_hubs/) |
| Collector | [loki.source.cloudflare](https://grafana.com/docs/alloy/latest/reference/components/loki.source.cloudflare/) |
| Collector | [loki.source.docker](https://grafana.com/docs/alloy/latest/reference/components/loki.source.docker/) |
| Collector | [loki.source.file](https://grafana.com/docs/alloy/latest/reference/components/loki.source.file/) |
| Collector | [loki.source.gcplog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gcplog/) |
| Collector | [loki.source.gelf](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gelf/) |
| Collector | [loki.source.heroku](https://grafana.com/docs/alloy/latest/reference/components/loki.source.heroku/) |
| Collector | [loki.source.journal](https://grafana.com/docs/alloy/latest/reference/components/loki.source.journal/) |
| Collector | [loki.source.kafka](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kafka/) |
| Collector | [loki.source.kubernetes](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes/) |
| Collector | [loki.source.kubernetes_events](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes_events/) |
| Collector | [loki.source.podlogs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.podlogs/) |
| Collector | [loki.source.syslog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.syslog/) |
| Collector | [loki.source.windowsevent](https://grafana.com/docs/alloy/latest/reference/components/loki.source.windowsevent/) |
| Collector | [otelcol.receiver.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.loki/) |
| Transformer| [loki.relabel](https://grafana.com/docs/alloy/latest/reference/components/loki.relabel/) |
| Transformer| [loki.process](https://grafana.com/docs/alloy/latest/reference/components/loki.process/) |
| Writer | [loki.write](https://grafana.com/docs/alloy/latest/reference/components/loki.write/) |
| Writer | [otelcol.exporter.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.loki/) |
| Writer | [otelcol.exporter.logging](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.logging/) |


## Interactive Tutorials

To learn more about how to configure Alloy to send logs to Loki within different scenarios, follow these interactive tutorials:

- [Sending OpenTelemetry logs to Loki using Alloy]({{< relref "./examples/alloy-otel-logs" >}})
- [Sending logs over Kafka to Loki using Alloy]({{< relref "./examples/alloy-kafka-logs" >}})


Loading

0 comments on commit 5fb0980

Please sign in to comment.