Skip to content

Commit

Permalink
docs: Update _index.md addressing #11575 (#11626)
Browse files Browse the repository at this point in the history
**What this PR does / why we need it**:
It improves the docs by addressing #11575 allowing users to use the
official fluent-bit helm chart over the deprecated grafana one

**Which issue(s) this PR fixes**:
Fixes #11575

**Special notes for your reviewer**:
@JStickler wanted to take a look at this.

**Checklist**
- [ x ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ x ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](0d4416a)

---------

Co-authored-by: J Stickler <[email protected]>
(cherry picked from commit 3b2278d)
  • Loading branch information
PeterStolz authored and grafana-delivery-bot[bot] committed Jan 11, 2024
1 parent 732e799 commit 061d88c
Showing 1 changed file with 49 additions and 11 deletions.
60 changes: 49 additions & 11 deletions docs/sources/send-data/fluentbit/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ weight: 500
---
# Fluent Bit client

[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the [Grafana Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki. You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines.
[Fluent Bit](https://fluentbit.io/) is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Fluent Bit Plugin described here or with the [Fluent-bit Loki output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/loki) to ship logs to Loki.
This plugin has more configuration options compared to the built-in Fluent Bit Loki plugin.
You can define which log files you want to collect using the [`Tail`](https://docs.fluentbit.io/manual/pipeline/inputs/tail) or [`Stdin`](https://docs.fluentbit.io/manual/pipeline/inputs/standard-input) data pipeline input. Additionally, Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc.) to structure and alter log lines.

## Usage

Expand Down Expand Up @@ -63,23 +65,59 @@ To ship logs from Docker containers to Grafana Cloud using Fluent Bit, you can u

You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.

To do so you can use our [Fluent Bit helm chart](https://github.com/grafana/helm-charts/tree/main/charts/fluent-bit):
To do so you can use the [Fluent Bit helm chart](https://github.com/fluent/helm-charts) with the following `values.yaml` changing the value of `FLUENT_LOKI_URL`:

```yaml
image:
# Here we use the Docker image which has the plugin installed
repository: grafana/fluent-bit-plugin-loki
tag: main-e2ed1c0

args:
- "-e"
- "/fluent-bit/bin/out_grafana_loki.so"
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/etc/conf/fluent-bit.conf

env:
# Note that for security reasons you should fetch the credentials through a Kubernetes Secret https://kubernetes.io/docs/concepts/configuration/secret/ . You may use the envFrom for this.
- name: FLUENT_LOKI_URL
value: https://user:[email protected]/loki/api/v1/push

config:
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
# Be aware that local clusters like docker-desktop or kind use the docker log format and not the cri (https://docs.fluentbit.io/manual/installation/kubernetes#container-runtime-interface-cri-parser)
multiline.parser docker, cri
Mem_Buf_Limit 5MB
Skip_Long_Lines On

outputs: |
[Output]
Name grafana-loki
Match kube.*
Url ${FLUENT_LOKI_URL}
Labels {job="fluent-bit"}
LabelKeys level,app # this sets the values for actual Loki streams and the other labels are converted to structured_metadata https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/
BatchWait 1
BatchSize 1001024
LineFormat json
LogLevel info
AutoKubernetesLabels true
```

```bash
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm upgrade --install fluent-bit grafana/fluent-bit \
--set loki.serviceName=loki.svc.cluster.local
helm install fluent-bit fluent/fluent-bit -f values.yaml
```

By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..).

Alternatively you can install the Loki and Fluent Bit all together using:

```bash
helm upgrade --install loki-stack grafana/loki-stack \
--set fluent-bit.enabled=true,promtail.enabled=false
```
If you also want to host your Loki instance inside the cluster install the [official Loki helm chart](https://grafana.com/docs/loki/latest/setup/install/helm/).

### AWS Elastic Container Service (ECS)

Expand Down

0 comments on commit 061d88c

Please sign in to comment.