Skip to content

Commit

Permalink
docs: move Storage page under Configuration (#12478)
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler authored Apr 5, 2024
1 parent b8a2e0c commit 3634ff4
Show file tree
Hide file tree
Showing 6 changed files with 25 additions and 24 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -11,24 +11,25 @@ is then compressed and stored in chunks in object stores such as S3 or GCS, or
even locally on the filesystem. A small index and highly compressed chunks
simplifies the operation and significantly lowers the cost of Loki.

Prior to Loki 2.0, chunks and index data were stored in separate backends:
object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data.
Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki.
More detailed information about TSDB can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/tsdb/).

Loki 2.0 brings an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store).
Loki 2.0 introduced an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store).
This type only requires one store, the object store, for both the index and chunks.
More detailed information can be found on the [operations page]({{< relref "../operations/storage/boltdb-shipper.md" >}}).
More detailed information about 'boltdb-shipper' can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/).

Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki.
Prior to Loki 2.0, chunks and index data were stored in separate backends:
object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. These "multistore" backends have been deprecated, as noted below.

You can find more storage information in the [manage section]({{< relref "../operations/storage/_index.md" >}}).
You can find more detailed information about all of the storage options in the [manage section](https://grafana.com/docs/loki/latest/operations/storage/).

## Single Store

Single Store refers to using object storage as the storage medium for both Loki's index as well as its data ("chunks"). There are two supported modes:

### TSDB (recommended)

Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
Starting in Loki 2.8, the [TSDB index store](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".

### BoltDB (deprecated)

Expand Down Expand Up @@ -90,7 +91,7 @@ This storage type for chunks is deprecated and may be removed in future major ve

### Cassandra (deprecated)

Cassandra can also be utilized for the index store and aside from the [boltdb-shipper]({{< relref "../operations/storage/boltdb-shipper" >}}), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
Cassandra can also be utilized for the index store and aside from the [boltdb-shipper](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.

{{< collapse title="Title of hidden content" >}}
This storage type for indexes is deprecated and may be removed in future major versions of Loki.
Expand Down
18 changes: 9 additions & 9 deletions docs/sources/operations/storage/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ weight:
---
# Manage storage

You can read a high level overview of Loki storage [here]({{< relref "../../storage/_index.md" >}})
You can read a high level overview of Loki storage [here](https://grafana.com/docs/loki/latest/configure/storage/)

Grafana Loki needs to store two different types of data: **chunks** and **indexes**.

Expand All @@ -18,21 +18,21 @@ format](#chunk-format) for how chunks are stored internally.
The **index** stores each stream's label set and links them to the individual
chunks.

Refer to Loki's [configuration]({{< relref "../../configure" >}}) for details on
Refer to Loki's [configuration](https://grafana.com/docs/loki/latest/configure/) for details on
how to configure the storage and the index.

For more information:

- [Table Manager]({{< relref "./table-manager" >}})
- [Retention]({{< relref "./retention" >}})
- [Logs Deletion]({{< relref "./logs-deletion" >}})
- [Table Manager](https://grafana.com/docs/loki/latest/operations/storage/table-manager/)
- [Retention](https://grafana.com/docs/loki/latest/operations/storage/retention/)
- [Logs Deletion](https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/)

## Supported Stores

The following are supported for the index:

- [TSDB]({{< relref "./tsdb" >}}) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store.
- [TSDB](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/) index store which stores boltdb index files in the object store.
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org)
Expand Down Expand Up @@ -76,7 +76,7 @@ When using S3 as object storage, the following permissions are needed:

Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`

See the [AWS deployment section]({{< relref "../../storage#aws-deployment-s3-single-store" >}}) on the storage page for a detailed setup guide.
See the [AWS deployment section](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store) on the storage page for a detailed setup guide.

### DynamoDB

Expand Down Expand Up @@ -134,7 +134,7 @@ Resources: `arn:aws:iam::<aws_account_id>:role/<role_name>`

When using IBM Cloud Object Storage (COS) as object storage, IAM `Writer` role is needed.

See the [IBM Cloud Object Storage section]({{< relref "../../storage#ibm-deployment-cos-single-store" >}}) on the storage page for a detailed setup guide.
See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki/latest/configure/storage/#ibm-deployment-cos-single-store) on the storage page for a detailed setup guide.

## Chunk Format

Expand Down
2 changes: 1 addition & 1 deletion docs/sources/release-notes/v2-9.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Grafana Labs is excited to announce the release of Loki 2.9.0 Here's a summary o
- **logfmt stage improvements**: logfmt parser now performs non-strict parsing by default which helps scan semi-structured log lines. [PR #9626](https://github.com/grafana/loki/pull/9626)

- **Deprecations**
- Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/storage/ for more information.
- Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/configure/storage.md for more information.
- The `table-manager` target is deprecated, because it is not used by "single store" implementations.
- The `-boltdb.shipper.compactor.*` CLI flags are deprecated in favor of `-compactor.*`.
- The `-ingester.unordered-writes` CLI flag is deprecated and will always default to `true` in the next major release.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ This guide assumes Loki will be installed in one of the modes above and that a `
**To grant access to S3 via an IAM role without providing credentials:**
1. Provision an IAM role, policy and S3 bucket as described in [Storage]({{< relref "../../../../storage#aws-deployment-s3-single-store" >}}).
1. Provision an IAM role, policy and S3 bucket as described in [Storage](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store).
- If the Terraform module was used note the annotation emitted by `terraform output -raw annotation`.

1. Add the IAM role annotation to the service account in `values.yaml`:
Expand Down
4 changes: 2 additions & 2 deletions docs/sources/setup/migrate/migrate-to-tsdb/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ keywords:
# Migrate to TSDB

[TSDB]({{< relref "../../../operations/storage/tsdb" >}}) is the recommended index type for Loki and is where the current development lies.
If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types]({{< relref "../../../storage#index-storage" >}}) that have been deprecated,
If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types](https://grafana.com/docs/loki/latest/configure/storage/#index-storage) that have been deprecated,
we strongly recommend migrating to TSDB.


### Configure TSDB index for an upcoming period

To begin the migration, add a new [period_config]({{< relref "../../../configure#period_config" >}}) entry in your [schema_config]({{< relref "../../../configure#schema_config" >}}).
You can read more about schema config [here]({{< relref "../../../storage#schema-config" >}}).
You can read more about schema config [here](https://grafana.com/docs/loki/latest/configure/storage/#schema-config).

{{% admonition type="note" %}}
You must roll out the new `period_config` change to all Loki components in order for it to take effect.
Expand Down
6 changes: 3 additions & 3 deletions docs/sources/setup/upgrade/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ period_config:
```

{{% admonition type="note" %}}
`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes]({{< relref "../../storage#index-storage" >}}).
`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes](https://grafana.com/docs/loki/latest/configure/storage/#index-storage).
{{% /admonition %}}

`path_prefix` defaults to `index/` which is same as the default value of the removed configurations.
Expand Down Expand Up @@ -205,7 +205,7 @@ All of these are cached to the `results_cache` which is configured in the `query

#### Write dedupe cache is deprecated
Write dedupe cache is deprecated because it not required by the newer single store indexes ([TSDB]({{< relref "../../operations/storage/tsdb" >}}) and [boltdb-shipper]({{< relref "../../operations/storage/boltdb-shipper" >}})).
If you using a [legacy index type]({{< relref "../../storage#index-storage" >}}), consider migrating to TSDB (recommended).
If you using a [legacy index type](https://grafana.com/docs/loki/latest/configure/storage/#index-storage), consider migrating to TSDB (recommended).

#### Embedded cache metric changes

Expand Down Expand Up @@ -1247,7 +1247,7 @@ schema_config:
④ Make sure this matches your existing config (e.g. maybe you were using gcs for your object_store)
⑤ 24h is required for boltdb-shipper
There are more examples on the [Storage description page]({{< relref "../../storage/_index.md#examples" >}}) including the information you need to setup the `storage` section for boltdb-shipper.
There are more examples on the [Storage description page](https://grafana.com/docs/loki/latest/configure/storage/#examples) including the information you need to setup the `storage` section for boltdb-shipper.


## 1.6.0
Expand Down

0 comments on commit 3634ff4

Please sign in to comment.