Skip to content

Commit

Permalink
docs: backporting #12508 to 30 branch
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler committed Apr 8, 2024
1 parent 01b693e commit 7802e5e
Show file tree
Hide file tree
Showing 3 changed files with 74 additions and 16 deletions.
26 changes: 12 additions & 14 deletions docs/sources/operations/storage/retention.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ If you have a lifecycle policy configured on the object store, please ensure tha
Granular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.

{{% admonition type="note" %}}
The Compactor does not support retention on [legacy index types]({{< relref "../../storage#index-storage" >}}). Please use the [Table Manager]({{< relref "./table-manager" >}}) when using legacy index types.
The Compactor does not support retention on [legacy index types](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/storage/#index-storage). Please use the [Table Manager](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/table-manager/) when using legacy index types.
Both the Table manager and legacy index types are deprecated and may be removed in future major versions of Loki.
{{% /admonition %}}

Expand Down Expand Up @@ -50,7 +50,7 @@ Chunks cannot be deleted immediately for the following reasons:

Marker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts.
{{% admonition type="note" %}}
We recommend running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.
Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.
{{% /admonition %}}

### Retention Configuration
Expand Down Expand Up @@ -100,7 +100,7 @@ Retention is only available if the index period is 24h. Single store TSDB and si

#### Configuring the retention period

Retention period is configured within the [`limits_config`]({{< relref "../../configure#limits_config" >}}) configuration section.
Retention period is configured within the [`limits_config`](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#limits_config) configuration section.

There are two ways of setting retention policies:

Expand Down Expand Up @@ -129,7 +129,7 @@ limits_config:
You can only use label matchers in the `selector` field of a `retention_stream` definition. Arbitrary LogQL expressions are not supported.
{{% /admonition %}}

Per tenant retention can be defined by configuring [runtime overrides]({{< relref "../../configure#runtime-configuration-file" >}}). For example:
Per tenant retention can be defined by configuring [runtime overrides](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#runtime-configuration-file). For example:

```yaml
overrides:
Expand All @@ -150,7 +150,7 @@ overrides:
```

Retention period for a given stream is decided based on the first match in this list:
1. If mutiple per-tenant `retention_stream` selectors match the stream, retention period with the highest priority is picked.
1. If multiple per-tenant `retention_stream` selectors match the stream, retention period with the highest priority is picked.
2. If multiple global `retention_stream` selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant `retention_stream` is set.
3. If a per-tenant `retention_period` is specified, it will be applied.
4. The global `retention_period` will be applied if none of the above match.
Expand Down Expand Up @@ -181,13 +181,13 @@ The example configurations defined above will result in the following retention

## Table Manager (deprecated)

Retention through the [Table Manager]({{< relref "./table-manager" >}}) is
Retention through the [Table Manager](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/table-manager/) is
achieved by relying on the object store TTL feature, and will work for both
[boltdb-shipper]({{< relref "./boltdb-shipper" >}}) store and chunk/index stores.
[boltdb-shipper](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/boltdb-shipper/) store and chunk/index stores.

In order to enable the retention support, the Table Manager needs to be
configured to enable deletions and a retention period. Please refer to the
[`table_manager`]({{< relref "../../configure#table_manager" >}})
[`table_manager`](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#table_manager)
section of the Loki configuration reference for all available options.
Alternatively, the `table-manager.retention-period` and
`table-manager.retention-deletes-enabled` command line flags can be used. The
Expand All @@ -196,15 +196,13 @@ can be parsed using the Prometheus common model [ParseDuration](https://pkg.go.d

{{% admonition type="warning" %}}
The retention period must be a multiple of the index and chunks table
`period`, configured in the [`period_config`]({{< relref "../../configure#period_config" >}})
block. See the [Table Manager]({{< relref "./table-manager#retention" >}}) documentation for
`period`, configured in the [`period_config`](https://grafana.com/docs/loki/latest/configure/#period_config) block.
See the [Table Manager](https://grafana.com/docs/loki/latest/operations/storage/table-manager/#retention) documentation for
more information.
{{% /admonition %}}

{{% admonition type="note" %}}
To avoid querying of data beyond the retention period,
`max_query_lookback` config in [`limits_config`]({{< relref "../../configure#limits_config" >}}) must be set to a value less than or equal to
what is set in `table_manager.retention_period`.
To avoid querying of data beyond the retention period,`max_query_lookback` config in [`limits_config`](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#limits_config) must be set to a value less than or equal to what is set in `table_manager.retention_period`.
{{% /admonition %}}

When using S3 or GCS, the bucket storing the chunks needs to have the expiry
Expand All @@ -225,7 +223,7 @@ intact; you will still be able to see related labels but will be unable to
retrieve the deleted log content.

For further details on the Table Manager internals, refer to the
[Table Manager]({{< relref "./table-manager" >}}) documentation.
[Table Manager](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/table-manager/) documentation.


## Example Configuration
Expand Down
3 changes: 1 addition & 2 deletions docs/sources/release-notes/next.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: V?.?
description: Version ?.? release notes
weight: 55
draft: true
---

# V?.?
Expand All @@ -14,5 +15,3 @@ Grafana Labs is excited to announce the release of Loki ?.?.? Here's a summary o
## Upgrade Considerations

## Bug fixes

- **Parse JSON String arrays properly so string elements can be retrieved**: [PR #11921](https://github.com/grafana/loki/pull/11921)]
61 changes: 61 additions & 0 deletions docs/sources/release-notes/v3-0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: v3.0
description: Version 3.0 release notes.
weight: 30
---

# v3.0

Grafana Labs and the Loki team are excited to announce the release of Loki 3.0. Here's a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the [CHANGELOG](https://github.com/grafana/loki/blob/release-3.0.0-rc1/CHANGELOG.md).

## Features and enhancements

Key features in Loki 3.0.0 include the following:

- **Query acceleration with Bloom filters** (experimental): This is designed to speed up filter queries, with best results for queries that are looking for a specific text string like an error message or UUID. For more information, refer to [Query acceleration with Blooms](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/query-acceleration-blooms/).

- **Native OpenTelemetry Support**: A simplified ingestion pipeline (Loki Exporter no longer needed) and a more intuitive query experience for OTel logs. For more information, refer to the [OTEL documentation](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/otel/).

- **Helm charts**: A major upgrade to the Loki helm chart introduces support for `Distributed` mode (also known as [microservices](https://grafana.com/docs/loki/<LOKI_VERSION>/get-started/deployment-modes/#microservices-mode) mode), includes memcached by default, and includes several updates to configurations to improve Loki operations.

- **Lambda/Promtail:** support dropping labels ([#10755](https://github.com/grafana/loki/issues/10755)) ([ec54c72](https://github.com/grafana/loki/commit/ec54c723ebbeeda88000dde188d539ecfe05dad8)).

- **Docs improvements**: All the Getting Started topics have been revised, including a new [Quickstart](https://grafana.com/docs/loki/latest/get-started/quick-start/) to help new users get up and running with Loki faster.The Storage, Configuration Reference, and API documentation have been updated to reflect deprecated and removed code, configuration options, and API endpoints.

## Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removes a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

- [Deprecated storage options](https://grafana.com/docs/loki/latest/storage/) including the deprecation of the BoltDB store.

- [Deprecated configuration options](https://grafana.com/docs/loki/next/configure/).

- [API endpoint deprecations](https://grafana.com/docs/loki/next/reference/api/#deprecated-endpoints).

To learn more about breaking changes in this release, refer to the [Upgrade guide](https://grafana.com/docs/loki/next/setup/upgrade/).

## Upgrade Considerations

The path from 2.9 to 3.0 includes several breaking changes. For important upgrade guideance, refer to the [Upgrade Guide](https://grafana.com/docs/loki/latest/setup/upgrade/) and the separate [Helm Upgrade Guide](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/).

## Bug fixes

### 3.0.0 (2024-05-08)
- All lifecycler cfgs ref a valid IPv6 addr and port combination ([#11121](https://github.com/grafana/loki/issues/11121)) ([6385b19](https://github.com/grafana/loki/commit/6385b195739bd7d4e9706faddd0de663d8e5331a))
- **deps:** update github.com/c2h5oh/datasize digest to 859f65c (main) ([#10820](https://github.com/grafana/loki/issues/10820)) ([c66ffd1](https://github.com/grafana/loki/commit/c66ffd125cd89f5845a75a1751186fa46d003f70))
- **deps:** update github.com/docker/go-plugins-helpers digest to 6eecb7b (main) ([#10826](https://github.com/grafana/loki/issues/10826)) ([fb9c496](https://github.com/grafana/loki/commit/fb9c496b21be62f56866ae0f92440085e7860a2a))
- **deps:** update github.com/grafana/gomemcache digest to 6947259 (main) ([#10836](https://github.com/grafana/loki/issues/10836)) ([2327789](https://github.com/grafana/loki/commit/2327789b5506d0ccc00d931195da17a2d47bf236))
- **deps:** update github.com/grafana/loki/pkg/push digest to 583aa28 (main) ([#10842](https://github.com/grafana/loki/issues/10842)) ([02d9418](https://github.com/grafana/loki/commit/02d9418270f4e615c1f78b0def635da7c0572ca4))
- **deps:** update github.com/grafana/loki/pkg/push digest to cfc4f0e (main) ([#10946](https://github.com/grafana/loki/issues/10946)) ([d27c4d2](https://github.com/grafana/loki/commit/d27c4d297dc6cce93ada98f16b962380ec933c6a))
- **deps:** update github.com/grafana/loki/pkg/push digest to e523809 (main) ([#11107](https://github.com/grafana/loki/issues/11107)) ([09cb9ae](https://github.com/grafana/loki/commit/09cb9ae76f4aef7dea477961c0c5424d7243bf2a))
- **deps:** update github.com/joncrlsn/dque digest to c2ef48c (main) ([#10947](https://github.com/grafana/loki/issues/10947)) ([1fe4885](https://github.com/grafana/loki/commit/1fe48858ae15b33646eedb85b05d6773a8bc5020))
- **deps:** update module google.golang.org/grpc [security] (main) ([#11031](https://github.com/grafana/loki/issues/11031)) ([0695424](https://github.com/grafana/loki/commit/0695424f7dd62435df3a9981276b40f3c5ef5641))
- **helm:** bump nginx-unprivilege to fix CVE ([#10754](https://github.com/grafana/loki/issues/10754)) ([dbf7dd4](https://github.com/grafana/loki/commit/dbf7dd4bac112a538a59907a8c6092504e7f4a91))
- **Parse JSON String arrays properly so string elements can be retrieved**: [PR #11921](https://github.com/grafana/loki/pull/11921)]
- **promtail:** correctly parse list of drop stage sources from YAML ([#10848](https://github.com/grafana/loki/issues/10848)) ([f51ee84](https://github.com/grafana/loki/commit/f51ee849b03c5f6b79f3e93cb7fd7811636bede2))
- **promtail:** prevent panic due to duplicate metric registration after reloaded ([#10798](https://github.com/grafana/loki/issues/10798)) ([47e2c58](https://github.com/grafana/loki/commit/47e2c5884f443667e64764f3fc3948f8f11abbb8))
- respect query matcher in ingester when getting label values ([#10375](https://github.com/grafana/loki/issues/10375)) ([85e2e52](https://github.com/grafana/loki/commit/85e2e52279ecac6dc111d5c113c54d6054d2c922))
- Sidecar configuration for Backend ([#10603](https://github.com/grafana/loki/issues/10603)) ([c29ba97](https://github.com/grafana/loki/commit/c29ba973a0b5b7b59613d210b741d5a547ea0e83))
- **tools/lambda-promtail:** Do not evaluate empty string for drop_labels ([#11074](https://github.com/grafana/loki/issues/11074)) ([94169a0](https://github.com/grafana/loki/commit/94169a0e6b5bf96426ad21e40f9583b721f35d6c))

0 comments on commit 7802e5e

Please sign in to comment.