Skip to content

Commit

Permalink
Merge branch 'release-3.0.x' into backport12508
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler authored Apr 8, 2024
2 parents 7802e5e + 0c9f219 commit 64a3dad
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 7 deletions.
12 changes: 6 additions & 6 deletions docs/sources/get-started/labels/structured-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: Describes how to enable structure metadata for logs and how to quer
# What is structured metadata

{{% admonition type="warning" %}}
Structured metadata was added to chunk format V4 which is used if the schema version is greater or equal to `13`. (See [Schema Config]({{< relref "../../storage#schema-config" >}}) for more details about schema versions. )
Structured metadata was added to chunk format V4 which is used if the schema version is greater or equal to `13`. (See [Schema Config](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/storage/#schema-config)) for more details about schema versions. )
{{% /admonition %}}

Selecting proper, low cardinality labels is critical to operating and querying Loki effectively. Some metadata, especially infrastructure related metadata, can be difficult to embed in log lines, and is too high cardinality to effectively store as indexed labels (and therefore reducing performance of the index).
Expand All @@ -29,12 +29,12 @@ It is an antipattern to extract information that already exists in your log line
## Attaching structured metadata to log lines

You have the option to attach structured metadata to log lines in the push payload along with each log line and the timestamp.
For more information on how to push logs to Loki via the HTTP endpoint, refer to the [HTTP API documentation]({{< relref "../../reference/api#ingest-logs" >}}).
For more information on how to push logs to Loki via the HTTP endpoint, refer to the [HTTP API documentation](https://grafana.com/docs/loki/<LOKI_VERSION>/reference/api/#ingest-logs).

Alternatively, you can use the Grafana Agent or Promtail to extract and attach structured metadata to your log lines.
See the [Promtail: Structured metadata stage]({{< relref "../../send-data/promtail/stages/structured_metadata" >}}) for more information.
See the [Promtail: Structured metadata stage](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/promtail/stages/structured_metadata/) for more information.

With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash]({{< relref "../../send-data/logstash/_index.md" >}}).
With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/logstash/).

{{% admonition type="warning" %}}
There are defaults for how much structured metadata can be attached per log line.
Expand All @@ -52,7 +52,7 @@ There are defaults for how much structured metadata can be attached per log line
## Querying structured metadata

Structured metadata is extracted automatically for each returned log line and added to the labels returned for the query.
You can use labels of structured metadata to filter log line using a [label filter expression]({{< relref "../../query/log_queries#label-filter-expression" >}}).
You can use labels of structured metadata to filter log line using a [label filter expression](https://grafana.com/docs/loki/<LOKI_VERSION>/query/log_queries/#label-filter-expression).

For example, if you have a label `pod` attached to some of your log lines as structured metadata, you can filter log lines using:

Expand All @@ -66,7 +66,7 @@ Of course, you can filter by multiple labels of structured metadata at the same
{job="example"} | pod="myservice-abc1234-56789" | trace_id="0242ac120002"
```

Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like `maximum of series (50000) reached for a single query`. You can use the [Keep]({{< relref "../../query/log_queries#keep-labels-expression" >}}) and [Drop]({{< relref "../../query/log_queries#drop-labels-expression" >}}) stages to filter out labels that you don't need.
Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like `maximum of series (50000) reached for a single query`. You can use the [Keep](https://grafana.com/docs/loki/<LOKI_VERSION>/query/log_queries/#keep-labels-expression) and [Drop](https://grafana.com/docs/loki/<LOKI_VERSION>/query/log_queries/#drop-labels-expression) stages to filter out labels that you don't need.
For example:

```logql
Expand Down
4 changes: 3 additions & 1 deletion docs/sources/operations/storage/retention.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,9 @@ more information.
{{% /admonition %}}

{{% admonition type="note" %}}
To avoid querying of data beyond the retention period,`max_query_lookback` config in [`limits_config`](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#limits_config) must be set to a value less than or equal to what is set in `table_manager.retention_period`.
To avoid querying of data beyond the retention period,
`max_query_lookback` config in [`limits_config`](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#limits_config) must be set to a value less than or equal to
what is set in `table_manager.retention_period`.
{{% /admonition %}}

When using S3 or GCS, the bucket storing the chunks needs to have the expiry
Expand Down

0 comments on commit 64a3dad

Please sign in to comment.