From 3634ff4d6497ef369e003cb49fefac0faa0f7359 Mon Sep 17 00:00:00 2001 From: J Stickler Date: Fri, 5 Apr 2024 14:42:17 -0400 Subject: [PATCH] docs: move Storage page under Configuration (#12478) --- .../_index.md => configure/storage.md} | 17 +++++++++-------- docs/sources/operations/storage/_index.md | 18 +++++++++--------- docs/sources/release-notes/v2-9.md | 2 +- .../install/helm/configure-storage/_index.md | 2 +- .../setup/migrate/migrate-to-tsdb/_index.md | 4 ++-- docs/sources/setup/upgrade/_index.md | 6 +++--- 6 files changed, 25 insertions(+), 24 deletions(-) rename docs/sources/{storage/_index.md => configure/storage.md} (93%) diff --git a/docs/sources/storage/_index.md b/docs/sources/configure/storage.md similarity index 93% rename from docs/sources/storage/_index.md rename to docs/sources/configure/storage.md index 2c967d2fc4b9..c14483e46dab 100644 --- a/docs/sources/storage/_index.md +++ b/docs/sources/configure/storage.md @@ -11,16 +11,17 @@ is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki. -Prior to Loki 2.0, chunks and index data were stored in separate backends: -object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. +Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. +More detailed information about TSDB can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/tsdb/). -Loki 2.0 brings an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store). +Loki 2.0 introduced an index mechanism named 'boltdb-shipper' and is what we now call [Single Store](#single-store). This type only requires one store, the object store, for both the index and chunks. -More detailed information can be found on the [operations page]({{< relref "../operations/storage/boltdb-shipper.md" >}}). +More detailed information about 'boltdb-shipper' can be found under the [manage section](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/). -Loki 2.8 adds TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. +Prior to Loki 2.0, chunks and index data were stored in separate backends: +object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. These "multistore" backends have been deprecated, as noted below. -You can find more storage information in the [manage section]({{< relref "../operations/storage/_index.md" >}}). +You can find more detailed information about all of the storage options in the [manage section](https://grafana.com/docs/loki/latest/operations/storage/). ## Single Store @@ -28,7 +29,7 @@ Single Store refers to using object storage as the storage medium for both Loki' ### TSDB (recommended) -Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper". +Starting in Loki 2.8, the [TSDB index store](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper". ### BoltDB (deprecated) @@ -90,7 +91,7 @@ This storage type for chunks is deprecated and may be removed in future major ve ### Cassandra (deprecated) -Cassandra can also be utilized for the index store and aside from the [boltdb-shipper]({{< relref "../operations/storage/boltdb-shipper" >}}), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering. +Cassandra can also be utilized for the index store and aside from the [boltdb-shipper](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering. {{< collapse title="Title of hidden content" >}} This storage type for indexes is deprecated and may be removed in future major versions of Loki. diff --git a/docs/sources/operations/storage/_index.md b/docs/sources/operations/storage/_index.md index 75740348d036..44f08d91bfb0 100644 --- a/docs/sources/operations/storage/_index.md +++ b/docs/sources/operations/storage/_index.md @@ -6,7 +6,7 @@ weight: --- # Manage storage -You can read a high level overview of Loki storage [here]({{< relref "../../storage/_index.md" >}}) +You can read a high level overview of Loki storage [here](https://grafana.com/docs/loki/latest/configure/storage/) Grafana Loki needs to store two different types of data: **chunks** and **indexes**. @@ -18,21 +18,21 @@ format](#chunk-format) for how chunks are stored internally. The **index** stores each stream's label set and links them to the individual chunks. -Refer to Loki's [configuration]({{< relref "../../configure" >}}) for details on +Refer to Loki's [configuration](https://grafana.com/docs/loki/latest/configure/) for details on how to configure the storage and the index. For more information: -- [Table Manager]({{< relref "./table-manager" >}}) -- [Retention]({{< relref "./retention" >}}) -- [Logs Deletion]({{< relref "./logs-deletion" >}}) +- [Table Manager](https://grafana.com/docs/loki/latest/operations/storage/table-manager/) +- [Retention](https://grafana.com/docs/loki/latest/operations/storage/retention/) +- [Logs Deletion](https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/) ## Supported Stores The following are supported for the index: -- [TSDB]({{< relref "./tsdb" >}}) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer. -- [Single Store (boltdb-shipper)]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store. +- [TSDB](https://grafana.com/docs/loki/latest/operations/storage/tsdb/) index store which stores TSDB index files in the object store. This is the recommended index store for Loki 2.8 and newer. +- [Single Store (boltdb-shipper)](https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/) index store which stores boltdb index files in the object store. - [Amazon DynamoDB](https://aws.amazon.com/dynamodb) - [Google Bigtable](https://cloud.google.com/bigtable) - [Apache Cassandra](https://cassandra.apache.org) @@ -76,7 +76,7 @@ When using S3 as object storage, the following permissions are needed: Resources: `arn:aws:s3:::`, `arn:aws:s3:::/*` -See the [AWS deployment section]({{< relref "../../storage#aws-deployment-s3-single-store" >}}) on the storage page for a detailed setup guide. +See the [AWS deployment section](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store) on the storage page for a detailed setup guide. ### DynamoDB @@ -134,7 +134,7 @@ Resources: `arn:aws:iam:::role/` When using IBM Cloud Object Storage (COS) as object storage, IAM `Writer` role is needed. -See the [IBM Cloud Object Storage section]({{< relref "../../storage#ibm-deployment-cos-single-store" >}}) on the storage page for a detailed setup guide. +See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki/latest/configure/storage/#ibm-deployment-cos-single-store) on the storage page for a detailed setup guide. ## Chunk Format diff --git a/docs/sources/release-notes/v2-9.md b/docs/sources/release-notes/v2-9.md index 414064331629..aebea6ce2644 100644 --- a/docs/sources/release-notes/v2-9.md +++ b/docs/sources/release-notes/v2-9.md @@ -26,7 +26,7 @@ Grafana Labs is excited to announce the release of Loki 2.9.0 Here's a summary o - **logfmt stage improvements**: logfmt parser now performs non-strict parsing by default which helps scan semi-structured log lines. [PR #9626](https://github.com/grafana/loki/pull/9626) - **Deprecations** - - Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/storage/ for more information. + - Legacy index and chunk stores that are not "single store" (such as `tsdb`, `boltdb-shipper`) are deprecated. These storage backends are Cassandra (`cassandra`), DynamoDB (`aws`, `aws-dynamo`), BigTable (`bigtable`, `bigtable-hashed`), GCP (`gcp`, `gcp-columnkey`), and gRPC (`grpc`). See https://grafana.com/docs/loki/latest/configure/storage.md for more information. - The `table-manager` target is deprecated, because it is not used by "single store" implementations. - The `-boltdb.shipper.compactor.*` CLI flags are deprecated in favor of `-compactor.*`. - The `-ingester.unordered-writes` CLI flag is deprecated and will always default to `true` in the next major release. diff --git a/docs/sources/setup/install/helm/configure-storage/_index.md b/docs/sources/setup/install/helm/configure-storage/_index.md index e51e8503fee0..169f6b160f40 100644 --- a/docs/sources/setup/install/helm/configure-storage/_index.md +++ b/docs/sources/setup/install/helm/configure-storage/_index.md @@ -38,7 +38,7 @@ This guide assumes Loki will be installed in one of the modes above and that a ` **To grant access to S3 via an IAM role without providing credentials:** -1. Provision an IAM role, policy and S3 bucket as described in [Storage]({{< relref "../../../../storage#aws-deployment-s3-single-store" >}}). +1. Provision an IAM role, policy and S3 bucket as described in [Storage](https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store). - If the Terraform module was used note the annotation emitted by `terraform output -raw annotation`. 1. Add the IAM role annotation to the service account in `values.yaml`: diff --git a/docs/sources/setup/migrate/migrate-to-tsdb/_index.md b/docs/sources/setup/migrate/migrate-to-tsdb/_index.md index 3407345b4acb..4d2d89547caf 100644 --- a/docs/sources/setup/migrate/migrate-to-tsdb/_index.md +++ b/docs/sources/setup/migrate/migrate-to-tsdb/_index.md @@ -11,14 +11,14 @@ keywords: # Migrate to TSDB [TSDB]({{< relref "../../../operations/storage/tsdb" >}}) is the recommended index type for Loki and is where the current development lies. -If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types]({{< relref "../../../storage#index-storage" >}}) that have been deprecated, +If you are running Loki with [boltb-shipper]({{< relref "../../../operations/storage/boltdb-shipper" >}}) or any of the [legacy index types](https://grafana.com/docs/loki/latest/configure/storage/#index-storage) that have been deprecated, we strongly recommend migrating to TSDB. ### Configure TSDB index for an upcoming period To begin the migration, add a new [period_config]({{< relref "../../../configure#period_config" >}}) entry in your [schema_config]({{< relref "../../../configure#schema_config" >}}). -You can read more about schema config [here]({{< relref "../../../storage#schema-config" >}}). +You can read more about schema config [here](https://grafana.com/docs/loki/latest/configure/storage/#schema-config). {{% admonition type="note" %}} You must roll out the new `period_config` change to all Loki components in order for it to take effect. diff --git a/docs/sources/setup/upgrade/_index.md b/docs/sources/setup/upgrade/_index.md index 4badb1a4c52a..879cbe999408 100644 --- a/docs/sources/setup/upgrade/_index.md +++ b/docs/sources/setup/upgrade/_index.md @@ -85,7 +85,7 @@ period_config: ``` {{% admonition type="note" %}} -`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes]({{< relref "../../storage#index-storage" >}}). +`path_prefix` only applies to TSDB and BoltDB indexes. This setting has no effect on [legacy indexes](https://grafana.com/docs/loki/latest/configure/storage/#index-storage). {{% /admonition %}} `path_prefix` defaults to `index/` which is same as the default value of the removed configurations. @@ -205,7 +205,7 @@ All of these are cached to the `results_cache` which is configured in the `query #### Write dedupe cache is deprecated Write dedupe cache is deprecated because it not required by the newer single store indexes ([TSDB]({{< relref "../../operations/storage/tsdb" >}}) and [boltdb-shipper]({{< relref "../../operations/storage/boltdb-shipper" >}})). -If you using a [legacy index type]({{< relref "../../storage#index-storage" >}}), consider migrating to TSDB (recommended). +If you using a [legacy index type](https://grafana.com/docs/loki/latest/configure/storage/#index-storage), consider migrating to TSDB (recommended). #### Embedded cache metric changes @@ -1247,7 +1247,7 @@ schema_config: ④ Make sure this matches your existing config (e.g. maybe you were using gcs for your object_store) ⑤ 24h is required for boltdb-shipper -There are more examples on the [Storage description page]({{< relref "../../storage/_index.md#examples" >}}) including the information you need to setup the `storage` section for boltdb-shipper. +There are more examples on the [Storage description page](https://grafana.com/docs/loki/latest/configure/storage/#examples) including the information you need to setup the `storage` section for boltdb-shipper. ## 1.6.0