Merge branch 'main' into 20230825-edit-document-level-permissions

_api-reference/nodes-apis/nodes-stats.md

@@ -731,7 +731,10 @@ Select the arrow to view the example response.
         "nxLWtMdXQmWA-ZBVWU8nwA": {
           "timestamp": 1698401391000,
           "cpu_utilization_percent": "0.1",
-          "memory_utilization_percent": "3.9"
+          "memory_utilization_percent": "3.9",
+          "io_usage_stats": {
+            "max_io_utilization_percent": "99.6"
+          }
         }
       },
       "admission_control": {
@@ -742,6 +745,14 @@ Select the arrow to view the example response.
               "indexing": 1
             }
           }
+        },
+        "global_io_usage": {
+          "transport": {
+            "rejection_count": {
+              "search": 3,
+              "indexing": 1
+            }
+          }
         }
       }
     }
@@ -1252,16 +1263,20 @@ The `resource_usage_stats` object contains the resource usage statistics. Each e
 Field | Field type | Description
 :--- |:-----------| :---
 timestamp | Integer    | The last refresh time for the resource usage statistics, in milliseconds since the epoch.
-cpu_utilization_percent | Float      | Statistics for the average CPU usage of OpenSearch process within the time period configured in the `node.resource.tracker.global_cpu_usage.window_duration` setting.
+cpu_utilization_percent | Float      | Statistics for the average CPU usage of any OpenSearch processes within the time period configured in the `node.resource.tracker.global_cpu_usage.window_duration` setting.
 memory_utilization_percent | Float      | The node JVM memory usage statistics within the time period configured in the `node.resource.tracker.global_jvmmp.window_duration` setting.
+max_io_utilization_percent  | Float     |  (Linux only) Statistics for the average IO usage of any OpenSearch processes within the time period configured in the `node.resource.tracker.global_io_usage.window_duration` setting.
 
 ### `admission_control`
 
 The `admission_control` object contains the rejection count of search and indexing requests based on resource consumption and has the following properties.
+
 Field | Field type | Description
 :--- | :--- | :---
-admission_control.global_cpu_usage.transport.rejection_count.search | Integer | The total number of search rejections in the transport layer when the node CPU usage limit was breached. In this case, additional search requests are rejected until the system recovers.
-admission_control.global_cpu_usage.transport.rejection_count.indexing | Integer | The total number of indexing rejections in the transport layer when the node CPU usage limit was breached. In this case, additional indexing requests are rejected until the system recovers.
+admission_control.global_cpu_usage.transport.rejection_count.search | Integer | The total number of search rejections in the transport layer when the node CPU usage limit was met. In this case, additional search requests are rejected until the system recovers. The CPU usage limit is configured in the `admission_control.search.cpu_usage.limit` setting.
+admission_control.global_cpu_usage.transport.rejection_count.indexing | Integer | The total number of indexing rejections in the transport layer when the node CPU usage limit was met. Any additional indexing requests are rejected until the system recovers. The CPU usage limit is configured in the `admission_control.indexing.cpu_usage.limit` setting.
+admission_control.global_io_usage.transport.rejection_count.search | Integer | The total number of search rejections in the transport layer when the node IO usage limit was met. Any additional search requests are rejected until the system recovers. The CPU usage limit is configured in the `admission_control.search.io_usage.limit` setting (Linux only).
+admission_control.global_io_usage.transport.rejection_count.indexing | Integer | The total number of indexing rejections in the transport layer when the node IO usage limit was met. Any additional indexing requests are rejected until the system recovers. The IO usage limit is configured in the `admission_control.indexing.io_usage.limit` setting (Linux only).
 
 ## Required permissions
 

_api-reference/snapshots/get-snapshot-status.md

@@ -29,9 +29,9 @@ Three request variants provide flexibility:
 
 * `GET _snapshot/_status` returns the status of all currently running snapshots in all repositories.
 
-* `GET _snapshot/<repository>/_status` returns the status of only currently running snapshots in the specified repository. This is the preferred variant.
+* `GET _snapshot/<repository>/_status` returns all currently running snapshots in the specified repository. This is the preferred variant.
 
-* `GET _snapshot/<repository>/<snapshot>/_status` returns the status of all snapshots in the specified repository whether they are running or not.
+* `GET _snapshot/<repository>/<snapshot>/_status` returns detailed status information for a specific snapshot in the specified repository, regardless of whether it's currently running or not. 
 
 Using the API to return state for other than currently running snapshots can be very costly for (1) machine machine resources and (2) processing time if running in the cloud. For each snapshot, each request causes file reads from all a snapshot's shards. 
 {: .warning}
@@ -420,4 +420,4 @@ All property values are Integers.
 :--- | :--- | :--- |
 | shards_stats | Object | See [Shard stats](#shard-stats). |
 | stats | Object | See [Snapshot file stats](#snapshot-file-stats). |
-| shards | list of Objects | List of objects containing information about the shards that include the snapshot. Properies of the shards are listed below in bold text. <br /><br /> **stage**: Current state of shards in the snapshot. Shard states are: <br /><br /> * DONE: Number of shards in the snapshot that were successfully stored in the repository. <br /><br /> * FAILURE: Number of shards in the snapshot that were not successfully stored in the repository. <br /><br /> * FINALIZE: Number of shards in the snapshot that are in the finalizing stage of being stored in the repository. <br /><br />* INIT: Number of shards in the snapshot that are in the initializing stage of being stored in the repository.<br /><br />* STARTED:  Number of shards in the snapshot that are in the started stage of being stored in the repository.<br /><br /> **stats**: See [Snapshot file stats](#snapshot-file-stats). <br /><br /> **total**: Total number and size of files referenced by the snapshot. <br /><br /> **start_time_in_millis**: Time (in milliseconds) when snapshot creation began. <br /><br /> **time_in_millis**: Total time (in milliseconds) that the snapshot took to complete.  |
+| shards | list of Objects | List of objects containing information about the shards that include the snapshot. OpenSearch returns the following properties about the shards. <br /><br /> **stage**: Current state of shards in the snapshot. Shard states are: <br /><br /> * DONE: Number of shards in the snapshot that were successfully stored in the repository. <br /><br /> * FAILURE: Number of shards in the snapshot that were not successfully stored in the repository. <br /><br /> * FINALIZE: Number of shards in the snapshot that are in the finalizing stage of being stored in the repository. <br /><br />* INIT: Number of shards in the snapshot that are in the initializing stage of being stored in the repository.<br /><br />* STARTED:  Number of shards in the snapshot that are in the started stage of being stored in the repository.<br /><br /> **stats**: See [Snapshot file stats](#snapshot-file-stats). <br /><br /> **total**: Total number and size of files referenced by the snapshot. <br /><br /> **start_time_in_millis**: Time (in milliseconds) when snapshot creation began. <br /><br /> **time_in_millis**: Total time (in milliseconds) that the snapshot took to complete.  |

_monitoring-your-cluster/metrics/getting-started.md

@@ -1,8 +1,9 @@
 ---
 layout: default
-title: Metrics framework 
-parent: Trace Analytics
-nav_order: 65
+title: Metrics framework
+nav_order: 1
+has_children: false
+has_toc: false
 redirect_from:
   - /monitoring-your-cluster/metrics/
 ---
@@ -95,3 +96,12 @@ The metrics framework feature supports various telemetry solutions through plugi
 2. **Exporters:** Exporters are responsible for persisting the data. OpenTelemetry provides several out-of-the-box exporters. OpenSearch supports the following exporters:
     - `LoggingMetricExporter`: Exports metrics to a log file, generating a separate file in the logs directory `_otel_metrics.log`. Default is `telemetry.otel.metrics.exporter.class=io.opentelemetry.exporter.logging.LoggingMetricExporter`.
     - `OtlpGrpcMetricExporter`: Exports spans through gRPC. To use this exporter, you need to install the `otel-collector` on the node. By default, it writes to the http://localhost:4317/ endpoint. To use this exporter, set the following static setting: `telemetry.otel.metrics.exporter.class=io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter`.
+  
+### Supported metric types
+
+The metrics framework feature supports the following metric types:
+
+1. **Counters:** Counters are continuous and synchronous meters used to track the frequency of events over time. Counters can only be incremented with positive values, making them ideal for measuring the number of monitoring occurrences such as errors, processed or received bytes, and total requests.
+2. **UpDown counters:** UpDown counters can be incremented with positive values or decremented with negative values. UpDown counters are well suited for tracking metrics like open connections, active requests, and other fluctuating quantities.
+3. **Histograms:** Histograms are valuable tools for visualizing the distribution of continuous data. Histograms offer insight into the central tendency, spread, skewness, and potential outliers that might exist in your metrics. Patterns such as normal distribution, skewed distribution, or bimodal distribution can be readily identified, making histograms ideal for analyzing latency metrics and assessing percentiles.
+4. **Asynchronous Gauges:** Asynchronous gauges capture the current value at the moment a metric is read. These metrics are non-additive and are commonly used to measure CPU utilization on a per-minute basis, memory utilization, and other real-time values.

_security-analytics/api-tools/alert-finding-api.md

@@ -149,13 +149,230 @@ You can specify the following parameters when getting findings.
 
 Parameter | Description 
 :--- | :---
-`detector_id` | The ID of the detector used to fetch alerts. Optional when the `detectorType` is specified. Otherwise required.
-`detectorType` | The type of detector used to fetch alerts. Optional when the `detector_id` is specified. Otherwise required.
+`detector_id` | The ID of the detector used to fetch alerts. Optional.
+`detectorType` | The type of detector used to fetch alerts. Optional.
 `sortOrder` | The order used to sort the list of findings. Possible values are `asc` or `desc`. Optional.
 `size` | An optional limit for the maximum number of results returned in the response. Optional.
+`startIndex` | The pagination indicator. Optional.
+`detectionType` |  The detection rule type that dictates the retrieval type for the findings. When the detection type is `threat`, it fetches threat intelligence feeds. When the detection type is `rule`, findings are fetched based on the detector's rule. Optional.
+`severity` |  The severity of the detector rule used to fetch alerts. Severity can be `critical`, `high`, `medium`, or `low`. Optional.
 
 ### Example request
 
+```json
+GET /_plugins/_security_analytics/findings/_search
+{
+  "total_findings": 2,
+  "findings": [
+    {
+      "detectorId": "b9ZN040Bjlggkcgx1d1W",
+      "id": "35efb736-c5d9-499d-b9b5-31f0a7d61251",
+      "related_doc_ids": [
+        "1"
+      ],
+      "index": "smallidx",
+      "queries": [
+        {
+          "id": "QdZN040Bjlggkcgxdd3X",
+          "name": "QdZN040Bjlggkcgxdd3X",
+          "fields": [],
+          "query": "field1: *value1*",
+          "tags": [
+            "high",
+            "ad_ldap"
+          ]
+        }
+      ],
+      "timestamp": 1708647166500,
+      "document_list": [
+        {
+          "index": "smallidx",
+          "id": "1",
+          "found": true,
+          "document": "{\n  \"field1\": \"value1\"\n}\n"
+        }
+      ]
+    },
+    {
+      "detectorId": "O9ZM040Bjlggkcgx6N1S",
+      "id": "a5022930-4503-4ca8-bf0a-320a2b1fb433",
+      "related_doc_ids": [
+        "1"
+      ],
+      "index": "smallidx",
+      "queries": [
+        {
+          "id": "KtZM040Bjlggkcgxkd04",
+          "name": "KtZM040Bjlggkcgxkd04",
+          "fields": [],
+          "query": "field1: *value1*",
+          "tags": [
+            "critical",
+            "ad_ldap"
+          ]
+        }
+      ],
+      "timestamp": 1708647166500,
+      "document_list": [
+        {
+          "index": "smallidx",
+          "id": "1",
+          "found": true,
+          "document": "{\n  \"field1\": \"value1\"\n}\n"
+        }
+      ]
+    }
+  ]
+}
+
+```
+
+```json
+GET /_plugins/_security_analytics/findings/_search?severity=high
+{
+    "total_findings": 1,
+    "findings": [
+        {
+            "detectorId": "b9ZN040Bjlggkcgx1d1W",
+            "id": "35efb736-c5d9-499d-b9b5-31f0a7d61251",
+            "related_doc_ids": [
+                "1"
+            ],
+            "index": "smallidx",
+            "queries": [
+                {
+                    "id": "QdZN040Bjlggkcgxdd3X",
+                    "name": "QdZN040Bjlggkcgxdd3X",
+                    "fields": [],
+                    "query": "field1: *value1*",
+                    "tags": [
+                        "high",
+                        "ad_ldap"
+                    ]
+                }
+            ],
+            "timestamp": 1708647166500,
+            "document_list": [
+                {
+                    "index": "smallidx",
+                    "id": "1",
+                    "found": true,
+                    "document": "{\n  \"field1\": \"value1\"\n}\n"
+                }
+            ]
+        }
+    ]
+}
+
+```
+
+```json
+GET /_plugins/_security_analytics/findings/_search?detectionType=rule
+{
+    "total_findings": 2,
+    "findings": [
+        {
+            "detectorId": "b9ZN040Bjlggkcgx1d1W",
+            "id": "35efb736-c5d9-499d-b9b5-31f0a7d61251",
+            "related_doc_ids": [
+                "1"
+            ],
+            "index": "smallidx",
+            "queries": [
+                {
+                    "id": "QdZN040Bjlggkcgxdd3X",
+                    "name": "QdZN040Bjlggkcgxdd3X",
+                    "fields": [],
+                    "query": "field1: *value1*",
+                    "tags": [
+                        "high",
+                        "ad_ldap"
+                    ]
+                }
+            ],
+            "timestamp": 1708647166500,
+            "document_list": [
+                {
+                    "index": "smallidx",
+                    "id": "1",
+                    "found": true,
+                    "document": "{\n  \"field1\": \"value1\"\n}\n"
+                }
+            ]
+        },
+        {
+            "detectorId": "O9ZM040Bjlggkcgx6N1S",
+            "id": "a5022930-4503-4ca8-bf0a-320a2b1fb433",
+            "related_doc_ids": [
+                "1"
+            ],
+            "index": "smallidx",
+            "queries": [
+                {
+                    "id": "KtZM040Bjlggkcgxkd04",
+                    "name": "KtZM040Bjlggkcgxkd04",
+                    "fields": [],
+                    "query": "field1: *value1*",
+                    "tags": [
+                        "critical",
+                        "ad_ldap"
+                    ]
+                }
+            ],
+            "timestamp": 1708647166500,
+            "document_list": [
+                {
+                    "index": "smallidx",
+                    "id": "1",
+                    "found": true,
+                    "document": "{\n  \"field1\": \"value1\"\n}\n"
+                }
+            ]
+        }
+    ]
+}
+
+
+```
+```json
+GET /_plugins/_security_analytics/findings/_search?detectionType=rule&severity=high
+{
+    "total_findings": 1,
+    "findings": [
+        {
+            "detectorId": "b9ZN040Bjlggkcgx1d1W",
+            "id": "35efb736-c5d9-499d-b9b5-31f0a7d61251",
+            "related_doc_ids": [
+                "1"
+            ],
+            "index": "smallidx",
+            "queries": [
+                {
+                    "id": "QdZN040Bjlggkcgxdd3X",
+                    "name": "QdZN040Bjlggkcgxdd3X",
+                    "fields": [],
+                    "query": "field1: *value1*",
+                    "tags": [
+                        "high",
+                        "ad_ldap"
+                    ]
+                }
+            ],
+            "timestamp": 1708647166500,
+            "document_list": [
+                {
+                    "index": "smallidx",
+                    "id": "1",
+                    "found": true,
+                    "document": "{\n  \"field1\": \"value1\"\n}\n"
+                }
+            ]
+        }
+    ]
+}
+
+```
+
 ```json
 GET /_plugins/_security_analytics/findings/_search?*detectorType*=
 {

_security/access-control/permissions.md

@@ -124,7 +124,7 @@ green open .kibana_3 XmTePICFRoSNf5O5uLgwRw 1 1 220 0 468.3kb 232.1kb
 
 ### Enabling system index permissions
 
-Users that have the permission [`restapi:admin/roles`]({{site.url}}{{site.baseurl}}/security/access-control/api/#access-control-for-the-api) are able to map system index permissions to all users in the same way they would for a cluster or index permission in the `roles.yml` file. However, to preserve some control over this permission, the `plugins.security.system_indices.permissions.enabled` setting allows you to enable or disable the system index permissions feature. This setting is disabled by default. To enable the system index permissions feature, set `plugins.security.system_indices.permissions.enabled` to `true`. For more information about this setting, see [Enabling user access to system indexes]({{site.url}}{{site.baseurl}}/security/configuration/yaml/#enabling-user-access-to-system-indexes).
+Users that have the permission [`restapi:admin/roles`]({{site.url}}{{site.baseurl}}/security/access-control/api/#access-control-for-the-api) are able to map system index permissions to all users in the same way they would for a cluster or index permission in the `roles.yml` file. However, to preserve some control over this permission, the `plugins.security.system_indices.permission.enabled` setting allows you to enable or disable the system index permissions feature. This setting is disabled by default. To enable the system index permissions feature, set `plugins.security.system_indices.permissions.enabled` to `true`. For more information about this setting, see [Enabling user access to system indexes]({{site.url}}{{site.baseurl}}/security/configuration/yaml/#enabling-user-access-to-system-indexes).
 
 Keep in mind that enabling this feature and mapping system index permissions to normal users gives those users access to indexes that may contain sensitive information and configurations essential to a cluster's health. We also recommend caution when mapping users to `restapi:admin/roles` because this permission gives a user not only the ability to assign the system index permission to another user but also the ability to self-assign access to any system index.
 {: .warning }