From 414bc413c653db2a980e664e101aef8fd4a2d0b6 Mon Sep 17 00:00:00 2001 From: J Stickler Date: Wed, 10 Apr 2024 11:06:10 -0400 Subject: [PATCH] docs: Manual backport of #12493 (#12557) --- docs/sources/alert/_index.md | 2 +- docs/sources/get-started/labels/bp-labels.md | 2 +- .../operations/storage/logs-deletion.md | 2 +- docs/sources/operations/storage/retention.md | 5 ++-- .../storage/table-manager/_index.md | 2 +- docs/sources/operations/storage/wal.md | 2 +- docs/sources/reference/_index.md | 2 +- .../reference/{api.md => loki-http-api.md} | 20 ++++++------- docs/sources/release-notes/v3-0.md | 6 ++-- docs/sources/send-data/fluentd/_index.md | 2 +- docs/sources/send-data/k6/_index.md | 12 ++++---- docs/sources/send-data/k6/query-scenario.md | 2 +- docs/sources/send-data/promtail/_index.md | 2 +- .../send-data/promtail/configuration.md | 2 +- docs/sources/setup/upgrade/_index.md | 28 ++----------------- .../setup/upgrade/upgrade-to-6x/index.md | 2 +- 16 files changed, 34 insertions(+), 59 deletions(-) rename docs/sources/reference/{api.md => loki-http-api.md} (98%) diff --git a/docs/sources/alert/_index.md b/docs/sources/alert/_index.md index 81a2671c2101..47bf991a565c 100644 --- a/docs/sources/alert/_index.md +++ b/docs/sources/alert/_index.md @@ -360,7 +360,7 @@ ruler: The Ruler supports the following types of storage: `azure`, `gcs`, `s3`, `swift`, `cos` and `local`. Most kinds of storage work with the sharded Ruler configuration in an obvious way, that is, configure all Rulers to use the same backend. -The local implementation reads the rule files off of the local filesystem. This is a read-only backend that does not support the creation and deletion of rules through the [Ruler API]({{< relref "../reference/api#ruler" >}}). Despite the fact that it reads the local filesystem this method can still be used in a sharded Ruler configuration if the operator takes care to load the same rules to every Ruler. For instance, this could be accomplished by mounting a [Kubernetes ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) onto every Ruler pod. +The local implementation reads the rule files off of the local filesystem. This is a read-only backend that does not support the creation and deletion of rules through the [Ruler API](https://grafana.com/docs/loki//reference/loki-http-api#ruler). Despite the fact that it reads the local filesystem this method can still be used in a sharded Ruler configuration if the operator takes care to load the same rules to every Ruler. For instance, this could be accomplished by mounting a [Kubernetes ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) onto every Ruler pod. A typical local configuration might look something like: ``` diff --git a/docs/sources/get-started/labels/bp-labels.md b/docs/sources/get-started/labels/bp-labels.md index c2bc2e925c23..7800345684ce 100644 --- a/docs/sources/get-started/labels/bp-labels.md +++ b/docs/sources/get-started/labels/bp-labels.md @@ -45,7 +45,7 @@ Loki has several client options: [Promtail]({{< relref "../../send-data/promtail Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied. Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality. -Series information can be queried through the [Series API]({{< relref "../../reference/api" >}}), or you can use [logcli]({{< relref "../../query" >}}). +Series information can be queried through the [Series API](https://grafana.com/docs/loki//reference/loki-http-api/), or you can use [logcli]({{< relref "../../query" >}}). In Loki 1.6.0 and newer the logcli series command added the `--analyze-labels` flag specifically for debugging high cardinality labels: diff --git a/docs/sources/operations/storage/logs-deletion.md b/docs/sources/operations/storage/logs-deletion.md index cb0337c7bc90..eef99609ffd3 100644 --- a/docs/sources/operations/storage/logs-deletion.md +++ b/docs/sources/operations/storage/logs-deletion.md @@ -11,7 +11,7 @@ Log entries that fall within a specified time window and match an optional line Log entry deletion is supported _only_ when TSDB or BoltDB shipper is configured as the index store. -The compactor component exposes REST [endpoints]({{< relref "../../reference/api#compactor" >}}) that process delete requests. +The compactor component exposes REST [endpoints](https://grafana.com/docs/loki//reference/loki-http-api#compactor) that process delete requests. Hitting the endpoint specifies the streams and the time window. The deletion of the log entries takes place after a configurable cancellation time period expires. diff --git a/docs/sources/operations/storage/retention.md b/docs/sources/operations/storage/retention.md index 553e408cefcc..dac44129217e 100644 --- a/docs/sources/operations/storage/retention.md +++ b/docs/sources/operations/storage/retention.md @@ -196,10 +196,9 @@ can be parsed using the Prometheus common model [ParseDuration](https://pkg.go.d {{% admonition type="warning" %}} The retention period must be a multiple of the index and chunks table -`period`, configured in the [`period_config`](https://grafana.com/docs/loki/latest/configure/#period_config) block. -See the [Table Manager](https://grafana.com/docs/loki/latest/operations/storage/table-manager/#retention) documentation for +`period`, configured in the [`period_config`](https://grafana.com/docs/loki//configure/#period_config) block. +See the [Table Manager](https://grafana.com/docs/loki//operations/storage/table-manager/#retention) documentation for more information. -{{% /admonition %}} {{% admonition type="note" %}} To avoid querying of data beyond the retention period, diff --git a/docs/sources/operations/storage/table-manager/_index.md b/docs/sources/operations/storage/table-manager/_index.md index 0e6ba42cc71f..625123f0bb0f 100644 --- a/docs/sources/operations/storage/table-manager/_index.md +++ b/docs/sources/operations/storage/table-manager/_index.md @@ -7,7 +7,7 @@ weight: # Table manager {{% admonition type="note" %}} -Table manager is only needed if you are using a multi-store [backend]({{< relref "../../storage" >}}). If you are using either TSDB (recommended), or BoltDB (deprecated) you do not need the Table Manager. +Table manager is only needed if you are using a multi-store [backend](https://grafana.com/docs/loki//configure/storage/). If you are using either TSDB (recommended), or BoltDB (deprecated) you do not need the Table Manager. {{% /admonition %}} Grafana Loki supports storing indexes and chunks in table-based data storages. When diff --git a/docs/sources/operations/storage/wal.md b/docs/sources/operations/storage/wal.md index 73034d3a76ac..2bf9010c948b 100644 --- a/docs/sources/operations/storage/wal.md +++ b/docs/sources/operations/storage/wal.md @@ -87,7 +87,7 @@ When scaling down, we must ensure existing data on the leaving ingesters are flu Consider you have 4 ingesters `ingester-0 ingester-1 ingester-2 ingester-3` and you want to scale down to 2 ingesters, the ingesters which will be shut down according to StatefulSet rules are `ingester-3` and then `ingester-2`. -Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the [`/ingester/shutdown?flush=true`]({{< relref "../../reference/api#flush-in-memory-chunks-and-shut-down" >}}) endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted. +Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the [`/ingester/shutdown?flush=true`](https://grafana.com/docs/loki//reference/loki-http-api#flush-in-memory-chunks-and-shut-down) endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted. After hitting the endpoint for `ingester-2 ingester-3`, scale down the ingesters to 2. diff --git a/docs/sources/reference/_index.md b/docs/sources/reference/_index.md index 89e8e38f7172..bfdcad72144a 100644 --- a/docs/sources/reference/_index.md +++ b/docs/sources/reference/_index.md @@ -9,4 +9,4 @@ weight: 1000 This section provides reference material for Loki. -1. [Loki HTTP API]({{< relref "./api" >}}) +- [Loki HTTP API](https://grafana.com/docs/loki//reference/loki-http-api/) diff --git a/docs/sources/reference/api.md b/docs/sources/reference/loki-http-api.md similarity index 98% rename from docs/sources/reference/api.md rename to docs/sources/reference/loki-http-api.md index 5682510e4d86..9348c9a3de60 100644 --- a/docs/sources/reference/api.md +++ b/docs/sources/reference/loki-http-api.md @@ -4,7 +4,7 @@ menuTitle: HTTP API description: Provides a reference page for the Loki HTTP API endpoints for data ingestion, data retrieval, and cluster management. aliases: - ../api/ -weight: 100 +weight: 500 --- # Loki HTTP API @@ -112,7 +112,7 @@ These HTTP endpoints are exposed by all individual components: ### Deprecated endpoints {{% admonition type="note" %}} -The following endpoints are deprecated. While they still exist and work, they should not be used for new deployments. +The following endpoints are deprecated. While they still exist and work, they should not be used for new deployments. Existing deployments should upgrade to use the supported endpoints. {{% /admonition %}} @@ -132,17 +132,17 @@ Existing deployments should upgrade to use the supported endpoints. Some Loki API endpoints return a result of a matrix, a vector, or a stream: - **Matrix**: a table of values where each row represents a different label set - and the columns are each sample values for that row over the queried time. - Matrix types are only returned when running a query that computes some value. +and the columns are each sample values for that row over the queried time. +Matrix types are only returned when running a query that computes some value. - **Instant Vector**: denoted in the type as just `vector`, an Instant Vector - represents the latest value of a calculation for a given labelset. Instant - Vectors are only returned when doing a query against a single point in - time. +represents the latest value of a calculation for a given labelset. Instant +Vectors are only returned when doing a query against a single point in +time. - **Stream**: a Stream is a set of all values (logs) for a given label set over the - queried time range. Streams are the only type that will result in log lines - being returned. +queried time range. Streams are the only type that will result in log lines +being returned. ### Timestamps @@ -270,7 +270,7 @@ GET /loki/api/v1/query This type of query is often referred to as an instant query. Instant queries are mostly used for metric type LogQL queries. It accepts the following query parameters in the URL: -- `query`: The [LogQL]({{< relref "../query" >}}) query to perform. Requests that do not use valid LogQL syntax will return errors. +- `query`: The [LogQL]({{< relref "../query" >}}) query to perform. Requests that do not use valid LogQL syntax will return errors. - `limit`: The max number of entries to return. It defaults to `100`. Only applies to query types which produce a stream (log lines) response. - `time`: The evaluation time for the query as a nanosecond Unix epoch or another [supported format](#timestamps). Defaults to now. - `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward`. diff --git a/docs/sources/release-notes/v3-0.md b/docs/sources/release-notes/v3-0.md index 0a9a0018b64e..68c89eec0ba9 100644 --- a/docs/sources/release-notes/v3-0.md +++ b/docs/sources/release-notes/v3-0.md @@ -22,13 +22,13 @@ Key features in Loki 3.0.0 include the following: - **Lambda/Promtail:** support dropping labels ([#10755](https://github.com/grafana/loki/issues/10755)) ([ec54c72](https://github.com/grafana/loki/commit/ec54c723ebbeeda88000dde188d539ecfe05dad8)). -- **Docs improvements**: All the Getting Started topics have been revised, including a new [Quickstart](https://grafana.com/docs/loki/latest/get-started/quick-start/) to help new users get up and running with Loki faster.The Storage, Configuration Reference, and API documentation have been updated to reflect deprecated and removed code, configuration options, and API endpoints. +- **Docs improvements**: All the Getting Started topics have been revised, including a new [Quickstart](https://grafana.com/docs/loki//get-started/quick-start/) to help new users get up and running with Loki faster.The Storage, Configuration Reference, and API documentation have been updated to reflect deprecated and removed code, configuration options, and API endpoints. ## Deprecations One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removes a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include: -- [Deprecated storage options](https://grafana.com/docs/loki/latest/storage/) including the deprecation of the BoltDB store. +- [Deprecated storage options](https://grafana.com/docs/loki//storage/) including the deprecation of the BoltDB store. - [Deprecated configuration options](https://grafana.com/docs/loki/next/configure/). @@ -38,7 +38,7 @@ To learn more about breaking changes in this release, refer to the [Upgrade guid ## Upgrade Considerations -The path from 2.9 to 3.0 includes several breaking changes. For important upgrade guideance, refer to the [Upgrade Guide](https://grafana.com/docs/loki/latest/setup/upgrade/) and the separate [Helm Upgrade Guide](https://grafana.com/docs/loki/latest/setup/upgrade/upgrade-to-6x/). +The path from 2.9 to 3.0 includes several breaking changes. For important upgrade guidance, refer to the [Upgrade Guide](https://grafana.com/docs/loki//setup/upgrade/) and the separate [Helm Upgrade Guide](https://grafana.com/docs/loki//setup/upgrade/upgrade-to-6x/). ## Bug fixes diff --git a/docs/sources/send-data/fluentd/_index.md b/docs/sources/send-data/fluentd/_index.md index a2dffcac2dc3..61195d04dda2 100644 --- a/docs/sources/send-data/fluentd/_index.md +++ b/docs/sources/send-data/fluentd/_index.md @@ -189,7 +189,7 @@ This plugin automatically adds a `fluentd_thread` label with the name of the buf ### `url` -The URL of the Loki server to send logs to. When sending data, the publish path (`../reference/api/loki/v1/push`) will automatically be appended. +The URL of the Loki server to send logs to. When sending data, the publish path (`../reference/api/loki-http-api/v1/push/`) will automatically be appended. By default the url is set to `https://logs-prod-us-central1.grafana.net`, the url of the Grafana Labs [hosted Loki](/products/cloud/) service. #### Proxy Support diff --git a/docs/sources/send-data/k6/_index.md b/docs/sources/send-data/k6/_index.md index 0bd3f6c995ff..44e2f15ce17a 100644 --- a/docs/sources/send-data/k6/_index.md +++ b/docs/sources/send-data/k6/_index.md @@ -84,12 +84,12 @@ The `Client` class exposes the following instance methods: | method | description | | ------ | ----------- | | `push()` | shortcut for `pushParameterized(5, 800*1024, 1024*1024)` | -| `pushParameterized(streams, minSize, maxSize)` | execute push request ([POST /loki/api/v1/push]({{< relref "../../reference/api#ingest-logs" >}})) | -| `instantQuery(query, limit)` | execute instant query ([GET /loki/api/v1/query]({{< relref "../../reference/api#query-logs-at-a-single-point-in-time" >}})) | -| `client.rangeQuery(query, duration, limit)` | execute range query ([GET /loki/api/v1/query_range]({{< relref "../../reference/api#query-logs-within-a-range-of-time" >}})) | -| `client.labelsQuery(duration)` | execute labels query ([GET /loki/api/v1/labels]({{< relref "../../reference/api#query-labels" >}})) | -| `client.labelValuesQuery(label, duration)` | execute label values query ([GET /loki/api/v1/label/\/values]({{< relref "../../reference/api#query-label-values" >}})) | -| `client.seriesQuery(matchers, duration)` | execute series query ([GET /loki/api/v1/series]({{< relref "../../reference/api#query-streams" >}})) | +| `pushParameterized(streams, minSize, maxSize)` | execute push request ([POST /loki/api/v1/push](https://grafana.com/docs/loki//reference/loki-http-api#ingest-logs) | +| `instantQuery(query, limit)` | execute instant query ([GET /loki/api/v1/query](https://grafana.com/docs/loki//reference/loki-http-api#query-logs-at-a-single-point-in-time) | +| `client.rangeQuery(query, duration, limit)` | execute range query ([GET /loki/api/v1/query_range](https://grafana.com/docs/loki//reference/loki-http-api#query-logs-within-a-range-of-time) | +| `client.labelsQuery(duration)` | execute labels query ([GET /loki/api/v1/labels](https://grafana.com/docs/loki//reference/loki-http-api#query-labels) | +| `client.labelValuesQuery(label, duration)` | execute label values query ([GET /loki/api/v1/label/\/values](https://grafana.com/docs/loki//reference/loki-http-api#query-label-values) | +| `client.seriesQuery(matchers, duration)` | execute series query ([GET /loki/api/v1/series](https://grafana.com/docs/loki//reference/loki-http-api#query-streams) | **Javascript load test example:** diff --git a/docs/sources/send-data/k6/query-scenario.md b/docs/sources/send-data/k6/query-scenario.md index 40a914079ae4..d5b8d38275d4 100644 --- a/docs/sources/send-data/k6/query-scenario.md +++ b/docs/sources/send-data/k6/query-scenario.md @@ -23,7 +23,7 @@ Loki has 5 types of queries: In a real-world use-case, such as querying Loki using it as a Grafana data source, all of these queries are used. Each of them has a different -[API]({{< relref "../../reference/api.md" >}}) endpoint. The xk6-loki extension +[API](https://grafana.com/docs/loki//reference/loki-http-api/) endpoint. The xk6-loki extension provides a [Javascript API](https://github.com/grafana/xk6-loki#javascript-api) for all these query types. diff --git a/docs/sources/send-data/promtail/_index.md b/docs/sources/send-data/promtail/_index.md index 720718ea1a2d..dd12165face6 100644 --- a/docs/sources/send-data/promtail/_index.md +++ b/docs/sources/send-data/promtail/_index.md @@ -106,7 +106,7 @@ Important details are: ## Loki Push API -Promtail can also be configured to receive logs from another Promtail or any Loki client by exposing the [Loki Push API]({{< relref "../../reference/api#ingest-logs" >}}) with the [loki_push_api]({{< relref "./configuration#loki_push_api" >}}) scrape config. +Promtail can also be configured to receive logs from another Promtail or any Loki client by exposing the [Loki Push API](https://grafana.com/docs/loki//reference/loki-http-api#ingest-logs) with the [loki_push_api](https://grafana.com/docs/loki//reference/loki-http-api#loki_push_api) scrape config. There are a few instances where this might be helpful: diff --git a/docs/sources/send-data/promtail/configuration.md b/docs/sources/send-data/promtail/configuration.md index cbd5c7fc717e..ce1e329c7ea0 100644 --- a/docs/sources/send-data/promtail/configuration.md +++ b/docs/sources/send-data/promtail/configuration.md @@ -926,7 +926,7 @@ max_message_length: ### loki_push_api -The `loki_push_api` block configures Promtail to expose a [Loki push API]({{< relref "../../reference/api#ingest-logs" >}}) server. +The `loki_push_api` block configures Promtail to expose a [Loki push API](https://grafana.com/docs/loki//reference/loki-http-api#ingest-logs) server. Each job configured with a `loki_push_api` will expose this API and will require a separate port. diff --git a/docs/sources/setup/upgrade/_index.md b/docs/sources/setup/upgrade/_index.md index fd5e2f470f3c..96390d5007bd 100644 --- a/docs/sources/setup/upgrade/_index.md +++ b/docs/sources/setup/upgrade/_index.md @@ -52,7 +52,6 @@ Here is the shortlist of things we think most people may encounter: * The `shared_store` config is removed. Refer to [Removed `shared_store` and `shared_store_key_prefix` from shipper configuration](#removed-shared_store-and-shared_store_key_prefix-from-shipper-configuration). * Loki now enforces a max line size of 256KB by default (you can disable this or increase this but this is how Grafana Labs runs Loki). Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). * Loki now enforces a max label limit of 15 labels per series, down from 30. Extra labels inflate the size of the index and reduce performance, you should almost never need more than 15 labels. Refer to [Changes to default configure values](#changes-to-default-configuration-values-in-30). - * Loki will automatically attempt to populate a `service_name` label on ingestion. Refer to [`service_name` label](#service_name-label). * There are many metric name changes. Refer to [Distributor metric changes](#distributor-metric-changes), [Embedded cache metric changes](#embedded-cache-metric-changes), and [Metrics namespace](#metrics-namespace). If you would like to see if your existing configuration will work with Loki 3.0: @@ -83,29 +82,6 @@ Structured Metadata is enabled by default in Loki 3.0, however, it requires your * Upgrade your index version and schema version before updating to 3.0, see [schema config upgrade](https://grafana.com/docs/loki//operations/storage/schema/). * Disable Structured Metadata (and therefor OTLP support) and upgrade to 3.0 and perform the schema migration after. This can be done by setting `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false`. -#### `service_name` label - -Loki 3.0 will automatically assign a `service_name` label to all ingested logs by default. A service name is something required by Open Telemetry semantic conventions and is something Grafana Labs is building into our future user interface and query experiences. - -Loki will attempt to create the `service_name` label by looking for the following labels in this order: - - - service - - app - - application - - name - - app_kubernetes_io_name - - container - - container_name - - component - - workload - - job - -If no label is found matching the list, a value of `unknown_service` is applied. - -You can change this list by providing a list of labels to `discover_service_name` in the [limits_config](/docs/loki//configure/#limits_config) block. - -**You can disable this by providing an empty value for `discover_service_name`.** - #### Removed `shared_store` and `shared_store_key_prefix` from shipper configuration The following CLI flags and the corresponding YAML settings to configure shared store for TSDB and BoltDB shippers are now removed: @@ -699,7 +675,7 @@ Alerting rules previously could be specified in two formats: 1.x format (legacy We decided to drop support for format `1.x` as it is fairly old and keeping support for it required a lot of code. In case you're still using the legacy format, take a look at -[Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) for instructions +[Alerting Rules](https://prometheus.io/docs/prometheus//configuration/alerting_rules/) for instructions on how to write alerting rules in the new format. For reference, the newer format follows a structure similar to the one below: @@ -1055,7 +1031,7 @@ In Loki 2.2 we changed the internal version of our chunk format from v2 to v3, t This makes it important to first upgrade to 2.0, 2.0.1, or 2.1 **before** upgrading to 2.2 so that if you need to rollback for any reason you can do so easily. {{% admonition type="note" %}} -2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and ugrade to 2.2, if you want to rollback, you must rollback to 2.0.1. +2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and upgrade to 2.2, if you want to rollback, you must rollback to 2.0.1. {{% /admonition %}} ### Loki Config diff --git a/docs/sources/setup/upgrade/upgrade-to-6x/index.md b/docs/sources/setup/upgrade/upgrade-to-6x/index.md index 48f4fde89062..ac57907e1f35 100644 --- a/docs/sources/setup/upgrade/upgrade-to-6x/index.md +++ b/docs/sources/setup/upgrade/upgrade-to-6x/index.md @@ -86,4 +86,4 @@ With these caches disabled, Loki will return to defaults which enables an in-mem #### Distributed mode -This chart introduces the ability to run Loki in distributed, or [microservices mode](https://grafana.com/docs/loki/latest/get-started/deployment-modes/#microservices-mode). Separate instructions on how to enable this as well as how to migrate from the existing community chart will be coming shortly. +This chart introduces the ability to run Loki in distributed, or [microservices mode](https://grafana.com/docs/loki//get-started/deployment-modes/#microservices-mode). Separate instructions on how to enable this as well as how to migrate from the existing community chart will be coming shortly.