Merge pull request #3995 from ItalyPaleAle/metrics-cardinality

[1.13] Update docs for low-cardinality metrics

daprdocs/content/en/operations/configuration/configuration-overview.md

@@ -106,21 +106,27 @@ The `metrics` section under the `Configuration` spec contains the following prop
 ```yml
 metrics:
   enabled: true
+  rules: []
+  http:
+    increasedCardinality: true
 ```
 
 The following table lists the properties for metrics:
 
 | Property     | Type   | Description |
 |--------------|--------|-------------|
-| `enabled` | boolean | Whether metrics should to be enabled. |
-| `rules`   | boolean | Named rule to filter metrics. Each rule contains a set of `labels` to filter on and a`regex`expression to apply to the metrics path. |
+| `enabled` | boolean | When set to true, the default, enables metrics collection and the metrics endpoint. |
+| `rules`   | array | Named rule to filter metrics. Each rule contains a set of `labels` to filter on and a `regex` expression to apply to the metrics path. |
+| `http.increasedCardinality` | boolean | When set to true, in the Dapr HTTP server each request path causes the creation of a new "bucket" of metrics. This can cause issues, including excessive memory consumption, when there many different requested endpoints (such as when interacting with RESTful APIs).<br>In Dapr 1.13 the default value is `true` (to preserve the behavior of Dapr <= 1.12), but will change to `false` in Dapr 1.14. |
 
-To mitigate high memory usage and egress costs associated with [high cardinality metrics]({{< ref "metrics-overview.md#high-cardinality-metrics" >}}), you can set regular expressions for every metric exposed by the Dapr sidecar. For example:
+To mitigate high memory usage and egress costs associated with [high cardinality metrics]({{< ref "metrics-overview.md#high-cardinality-metrics" >}}) with the HTTP server, you should set the `metrics.http.increasedCardinality` property to `false`.
+
+Using rules, you can set regular expressions for every metric exposed by the Dapr sidecar. For example:
 
 ```yml
-metric:
-    enabled: true
-    rules:
+metrics:
+  enabled: true
+  rules:
     - name: dapr_runtime_service_invocation_req_sent_total
       labels:
       - name: method

daprdocs/content/en/operations/observability/metrics/metrics-overview.md

@@ -3,22 +3,24 @@ type: docs
 title: "Configure metrics"
 linkTitle: "Overview"
 weight: 4000
-description: "Enable or disable Dapr metrics "
+description: "Enable or disable Dapr metrics"
 ---
 
 By default, each Dapr system process emits Go runtime/process metrics and has their own [Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
 
 ## Prometheus endpoint
-The Dapr sidecars exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
+
+The Dapr sidecar exposes a [Prometheus](https://prometheus.io/)-compatible metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
 
 ## Configuring metrics using the CLI
 
 The metrics application endpoint is enabled by default. You can disable it by passing the command line argument `--enable-metrics=false`.
 
-The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to Daprd. 
+The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to daprd.
 
 ## Configuring metrics in Kubernetes
-You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, `daprd` does not open the metrics listening port.
+
+You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, daprd does not open the metrics listening port.
 
 The following Kubernetes deployment example shows how metrics are explicitly enabled with the port specified as "9090".
 
@@ -54,10 +56,8 @@ spec:
 ```
 
 ## Configuring metrics using application configuration
-You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars running in a specific namespace:
 
-- Use the `metrics` spec configuration.
-- Set `enabled: false` to disable the metrics in the Dapr runtime.
+You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars by default, set `spec.metrics.enabled` to `false`.
 
 ```yaml
 apiVersion: dapr.io/v1alpha1
@@ -66,42 +66,46 @@ metadata:
   name: tracing
   namespace: default
 spec:
-  tracing:
-    samplingRate: "1"
   metrics:
     enabled: false
 ```
 
 ## High cardinality metrics
 
-Depending on your use case, some metrics emitted by Dapr might contain values that have a high cardinality. This might cause increased memory usage for the Dapr process/container and incur expensive egress costs in certain cloud environments. To mitigate this issue, you can set regular expressions for every metric exposed by the Dapr sidecar. [See a list of all Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
+When invoking Dapr using HTTP, the legacy behavior (and current default as of Dapr 1.13) is to create a separate "bucket" for each requested method. When working with RESTful APIs, this can cause very high cardinality, with potential negative impact on memory usage and CPU.
+
+Dapr 1.13 introduces a new option for the Dapr Configuration resource `spec.metrics.http.increasedCardinality`: when set to `false`, it reports metrics for the HTTP server for each "abstract" method (for example, requesting from a state store) instead of creating a "bucket" for each concrete request path.
+
+The default value of `spec.metrics.http.increasedCardinality` is `true` in Dapr 1.13, to maintain the same behavior as Dapr 1.12 and older. However, the value will change to `false` (low-cardinality metrics by default) in Dapr 1.14.
+
+Setting `spec.metrics.http.increasedCardinality` to `false` is **recommended** to all Dapr users, to reduce resource consumption. The pre-1.13 behavior, which is used when the option is `true`, is considered legacy and is only maintained for users who have special requirements around backwards-compatibility.
+
+## Transform metrics with regular expressions
 
-The following example shows how to apply a regular expression for the label `method` in the metric `dapr_runtime_service_invocation_req_sent_total`:
+You can set regular expressions for every metric exposed by the Dapr sidecar to "transform" their values. [See a list of all Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
+
+The name of the rule must match the name of the metric that is transformed. The following example shows how to apply a regular expression for the label `method` in the metric `dapr_runtime_service_invocation_req_sent_total`:
 
 ```yaml
 apiVersion: dapr.io/v1alpha1
 kind: Configuration
 metadata:
   name: daprConfig
 spec:
-  metric:
-      enabled: true
-      rules:
+  metrics:
+    enabled: true
+    increasedCardinality: true
+    rules:
       - name: dapr_runtime_service_invocation_req_sent_total
         labels:
         - name: method
           regex:
             "orders/": "orders/.+"
 ```
 
-When this configuration is applied, a recorded metric with the `method` label of `orders/a746dhsk293972nz` will be replaced with `orders/`.
-
-### Watch the demo
-
-Watch [this video to walk through handling high cardinality metrics](https://youtu.be/pOT8teL6j_k?t=1524):
-
-<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/pOT8teL6j_k?start=1524" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+When this configuration is applied, a recorded metric with the `method` label of `orders/a746dhsk293972nz` is replaced with `orders/`.
 
+Using regular expressions to reduce metrics cardinality is considered legacy. We encourage all users to set `spec.metrics.http.increasedCardinality` to `false` instead, which is simpler to configure and offers better performance.
 
 ## References
 

daprdocs/content/en/reference/resource-specs/configuration-schema.md

@@ -27,8 +27,17 @@ spec:
     stdout: true
     otel:
       endpointAddress: <REPLACE-WITH-ENDPOINT-ADDRESS>
-      isSecure: false
+      isSecure: <TRUE-OR-FALSE>
       protocol: <HTTP-OR-GRPC>
+  metrics:
+    enabled: <TRUE-OR-FALSE>
+    rules:
+      - name: <METRIC-NAME>
+        labels:
+          - name: <LABEL-NAME>
+            regex: {}
+    http:
+      increasedCardinality: <TRUE-OR-FALSE>
   httpPipeline: # for incoming http calls
     handlers:
       - name: <HANDLER-NAME>