Add documentation for OCSF field mapping and correlation engine API (#…

…4549) * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * add missing param descriptions (#4555) Signed-off-by: Subhobrata Dey <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit c6db296. merge conflict with same edits on another branch# * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * Revert "fix#4500 ocsf fields and api" This reverts commit 7a70dc3. merge conflict with second branch :wq * Revert "fix#4500 ocsf fields and api" This reverts commit ef13a74. * fix#4500 edits post merge conflict Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> * fix#4500 ocsf fields and api Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>
opensearch-project · Dec 20, 2023 · 32389c6 · 32389c6
1 parent a63b578
commit 32389c6
Show file tree

Hide file tree

Showing 7 changed files with 248 additions and 2 deletions.
diff --git a/_security-analytics/api-tools/correlation-eng.md b/_security-analytics/api-tools/correlation-eng.md
@@ -0,0 +1,224 @@
+---
+layout: default
+title: Correlation engine APIs
+parent: API tools
+nav_order: 55
+---
+
+
+# Correlation engine APIs
+
+Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain time window, and perform other tasks.
+
+
+## Create correlation rules between log types
+
+This API is used to create correlation rules:
+
+```json
+POST /_plugins/_security_analytics/correlation/rules
+```
+
+### Request fields
+
+| Field | Type | Description |
+| :--- | :--- |:--- |
+| `index` | String | The name of the index used as the log source. |
+| `query` | String | The query used to filter security logs for correlation. |
+| `category` | String | The log type associated with the log source. |
+
+#### Example request
+
+```json
+POST /_plugins/_security_analytics/correlation/rules
+{
+  "correlate": [
+    {
+      "index": "vpc_flow",
+      "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6",
+      "category": "network"
+    },
+    {
+      "index": "windows",
+      "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*",
+      "category": "windows"
+    },
+    {
+      "index": "ad_logs",
+      "query": "ResultType:50126",
+      "category": "ad_ldap"
+    },
+    {
+      "index": "app_logs",
+      "query": "endpoint:/customer_records.txt",
+      "category": "others_application"
+    }
+  ]
+}
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+  "_id": "DxKEUIkBpIjg64IK4nXg",
+  "_version": 1,
+  "rule": {
+    "name": null,
+    "correlate": [
+      {
+        "index": "vpc_flow",
+        "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6",
+        "category": "network"
+      },
+      {
+        "index": "windows",
+        "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*",
+        "category": "windows"
+      },
+      {
+        "index": "ad_logs",
+        "query": "ResultType:50126",
+        "category": "ad_ldap"
+      },
+      {
+        "index": "app_logs",
+        "query": "endpoint:/customer_records.txt",
+        "category": "others_application"
+      }
+    ]
+  }
+}
+```
+
+### Response fields
+
+| Field | Type | Description |
+| :--- | :--- |:--- |
+| `_id` | String | The Id for the new rule. |
+
+
+## List all findings and their correlations within a time window
+
+This API provides a list of all findings and their correlations within a specified time window:
+
+```json
+GET /_plugins/_security_analytics/correlations?start_timestamp=<start time in milliseconds>&end_timestamp=<end time in milliseconds>
+```
+
+### Query parameters
+
+| Parameter | Type | Description |
+| :--- | :--- |:--- |
+| `start_timestamp` | Number | Start time for the time window, in milliseconds. |
+| `end_timestamp` | Number | End time for the time window, in milliseconds. |
+
+#### Example request
+
+```json
+GET /_plugins/_security_analytics/correlations?start_timestamp=1689289210000&end_timestamp=1689300010000
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+  "findings": [
+    {
+      "finding1": "931de5f0-a276-45d5-9cdb-83e1045a3630",
+      "logType1": "network",
+      "finding2": "1e6f6a12-83f1-4a38-9bb8-648f196859cc",
+      "logType2": "test_windows",
+      "rules": [
+        "nqI2TokBgL5wWFPZ6Gfu"
+      ]
+    }
+  ]
+}
+```
+
+### Response fields
+
+| Field | Type | Description |
+| :--- | :--- |:--- |
+| `finding1` | String | The Id for a first finding in the correlation. |
+| `logType1` | String | The log type associated with the first finding. |
+| `finding2` | String | The Id for a second finding in the correlation. |
+| `logType2` | String | The log type associated with the second finding. |
+| `rules` | Array | A list of correlation rule IDs associated with the correlated findings. |
+
+
+## List correlations for a finding belonging to a log type
+
+This API is used to list correlations for specific findings and the log types associated with them:
+
+```json
+GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m
+```
+
+### Query parameters
+
+| Parameter | Type | Description |
+| :--- | :--- |:--- |
+| `finding` | String | The finding ID. |
+| `detector_type` | String | The log type for the detector. |
+| `nearby_findings` | Number | The number of nearby findings with respect to the given finding Id. |
+| `time_window` | String | Sets a time window in which all of the correlations must have occurred together. |
+
+
+#### Example request
+
+```json
+GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+```json
+{
+  "findings": [
+    {
+      "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5",
+      "detector_type": "others_application",
+      "score": 0.000015182109564193524
+    },
+    {
+      "finding": "2485b623-6573-42f4-a055-9b927e38a65f",
+      "detector_type": "ad_ldap",
+      "score": 0.000001615897872397909
+    },
+    {
+      "finding": "051e00ad-5996-4c41-be20-f992451d1331",
+      "detector_type": "windows",
+      "score": 0.000016230604160227813
+    },
+    {
+      "finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b",
+      "detector_type": "s3",
+      "score": 0.000001759401811796124
+    },
+    {
+      "finding": "9b86980e-5fb7-4c5a-bd1b-879a1e3baf12",
+      "detector_type": "network",
+      "score": 0.0000016306962606904563
+    },
+    {
+      "finding": "e7dea5a1-164f-48f9-880e-4ba33e508713",
+      "detector_type": "network",
+      "score": 0.00001632626481296029
+    }
+  ]
+}
+```
+
+### Response fields
+
+| Field | Type | Description |
+| :--- | :--- |:--- |
+| `finding` | String | The finding ID. |
+| `detector_type` | String | The log type associated with the finding. |
+| `score` | Number | The correlation score for the correlated finding. The score is based on the proximity of relevant findings in the threat scenario defined by the correlation rule. |
+
diff --git a/_security-analytics/api-tools/detector-api.md b/_security-analytics/api-tools/detector-api.md
@@ -26,7 +26,7 @@ Field | Type | Description
 `enabled` | Boolean | Sets the detector as either active (true) or inactive (false). Default is `true` when a new detector is created. Required.
 `name` | String | Name of the detector. Name should only consist of upper and lowercase letters, numbers 0-9, hyphens, spaces, and underscores. Use between 5 and 50 characters. Required.
 `detector_type` | String | The log type that defines the detector. Options are `linux`, `network` ,`windows`, `ad_ldap`, `apache_access`, `cloudtrail`, `dns`, and `s3`. Required.
-`schedule` | Object | The schedule that determines how often the detector runs. For information on specifying fixed intervals in the API, see [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/).
+`schedule` | Object | The schedule that determines how often the detector runs. For information about specifying fixed intervals in the API, see the [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/).
 `schedule.period` | Object | Details for the frequency of the schedule.
 `schedule.period.interval` | Integer | The interval at which the detector runs.
 `schedule.period.unit` | String | The interval's unit of time.

diff --git a/_security-analytics/api-tools/index.md b/_security-analytics/api-tools/index.md
@@ -18,4 +18,5 @@ The APIs for Security Analytics are separated into the following categories:
 * [Rules APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/rule-api/)
 * [Mappings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/)
 * [Alerts and findings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/alert-finding-api/)
+* [Correlation engine APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/correlation-eng/)
 
diff --git a/_security-analytics/api-tools/mappings-api.md b/_security-analytics/api-tools/mappings-api.md
@@ -11,6 +11,18 @@ The following APIs can be used for a number of tasks related to mappings, from c
 
 ## Get Mappings View
 
+
+this API returns a view of the fields contained in an index used as a log source.
+
+### Request fields
+
+The following fields are used to get field mappings.
+
+Field | Type | Description
+:--- | :--- |:--- |
+| `index_name` | String | The name of the index used for log ingestion. |
+| `rule_topic` | String | The log type of the index. |
+
 ### Example request
 
 ```json

diff --git a/_security-analytics/api-tools/rule-api.md b/_security-analytics/api-tools/rule-api.md
@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear
 
 ## Create Custom Rule
 
-The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
+The Create Custom Rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
 
 ```json
 POST /_plugins/_security_analytics/rules?category=windows

diff --git a/_security-analytics/sec-analytics-config/detectors-config.md b/_security-analytics/sec-analytics-config/detectors-config.md
@@ -60,6 +60,15 @@ Security Analytics takes advantage of prepackaged Sigma rules for security event
 
 Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.
 
+#### Amazon Security Lake logs
+
+[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
+
+The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
+
+To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
+{: .tip }
+
 ### Automatically mapped fields
 
 Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Expand **Automatically mapped fields** to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.

diff --git a/images/Security/detector_rules.png b/images/Security/detector_rules.png