Add documentation for OCSF field mapping and correlation engine API #4549
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,224 @@ | ||||||
| --- | ||||||
| layout: default | ||||||
| title: Correlation engine APIs | ||||||
| parent: API tools | ||||||
| nav_order: 55 | ||||||
| --- | ||||||
|
|
||||||
|
|
||||||
| # Correlation engine APIs | ||||||
|
|
||||||
| Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks. | ||||||
|
|
||||||
|
|
||||||
| ## Create correlation rules between log types | ||||||
|
|
||||||
| Used to create correlation rules. | ||||||
cwillum marked this conversation as resolved.
Show resolved
Hide resolved
cwillum marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ```json | ||||||
| POST /_plugins/_security_analytics/correlation/rules | ||||||
| ``` | ||||||
|
|
||||||
| ### Request fields | ||||||
|
|
||||||
| | Field | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `index` | String | The name of the index used as the log source. | | ||||||
| | `query` | String | The query used to filter security logs for correlation. | | ||||||
| | `category` | String | The log type associated with the log source. | | ||||||
|
|
||||||
| #### Example request | ||||||
|
|
||||||
| ```json | ||||||
| POST /_plugins/_security_analytics/correlation/rules | ||||||
| { | ||||||
| "correlate": [ | ||||||
| { | ||||||
| "index": "vpc_flow", | ||||||
| "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6", | ||||||
| "category": "network" | ||||||
| }, | ||||||
| { | ||||||
| "index": "windows", | ||||||
| "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*", | ||||||
| "category": "windows" | ||||||
| }, | ||||||
| { | ||||||
| "index": "ad_logs", | ||||||
| "query": "ResultType:50126", | ||||||
| "category": "ad_ldap" | ||||||
| }, | ||||||
| { | ||||||
| "index": "app_logs", | ||||||
| "query": "endpoint:/customer_records.txt", | ||||||
| "category": "others_application" | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
| {% include copy-curl.html %} | ||||||
|
|
||||||
| #### Example response | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "_id": "DxKEUIkBpIjg64IK4nXg", | ||||||
| "_version": 1, | ||||||
| "rule": { | ||||||
| "name": null, | ||||||
| "correlate": [ | ||||||
| { | ||||||
| "index": "vpc_flow", | ||||||
| "query": "dstaddr:4.5.6.7 or dstaddr:4.5.6.6", | ||||||
| "category": "network" | ||||||
| }, | ||||||
| { | ||||||
| "index": "windows", | ||||||
| "query": "winlog.event_data.SubjectDomainName:NTAUTHORI*", | ||||||
| "category": "windows" | ||||||
| }, | ||||||
| { | ||||||
| "index": "ad_logs", | ||||||
| "query": "ResultType:50126", | ||||||
| "category": "ad_ldap" | ||||||
| }, | ||||||
| { | ||||||
| "index": "app_logs", | ||||||
| "query": "endpoint:/customer_records.txt", | ||||||
| "category": "others_application" | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| | Field | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `_id` | String | The new rule's ID. | | ||||||
cwillum marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
|
|
||||||
| ## List all findings and their correlations within a time window | ||||||
|
|
||||||
| Provides a list of all findings and their correlations within a specified window of time. | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Please rephrase as a complete sentence naming the noun (what is used to do this?). |
||||||
|
|
||||||
| ```json | ||||||
| GET /_plugins/_security_analytics/correlations?start_timestamp=<start time in milliseconds>&end_timestamp=<end time in milliseconds> | ||||||
| ``` | ||||||
|
|
||||||
| ### Query parameters | ||||||
|
|
||||||
| | Parameter | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `start_timestamp` | Number | Start time for the time window in milliseconds. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| | `end_timestamp` | Number | End time for the time window in milliseconds. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| #### Example request | ||||||
|
|
||||||
| ```json | ||||||
| GET /_plugins/_security_analytics/correlations?start_timestamp=1689289210000&end_timestamp=1689300010000 | ||||||
| ``` | ||||||
| {% include copy-curl.html %} | ||||||
|
|
||||||
| #### Example response | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "findings": [ | ||||||
| { | ||||||
| "finding1": "931de5f0-a276-45d5-9cdb-83e1045a3630", | ||||||
| "logType1": "network", | ||||||
| "finding2": "1e6f6a12-83f1-4a38-9bb8-648f196859cc", | ||||||
| "logType2": "test_windows", | ||||||
| "rules": [ | ||||||
| "nqI2TokBgL5wWFPZ6Gfu" | ||||||
| ] | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| | Field | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `finding1` | String | The ID for a first finding in the correlation. | | ||||||
| | `logType1` | String | The log type associated with the first finding. | | ||||||
| | `finding2` | String | The ID for a second finding in the correlation. | | ||||||
| | `logType2` | String | The log type associated with the second finding. | | ||||||
| | `rules` | Array | A list of correlation rule IDs associated with the correlated findings. | | ||||||
|
|
||||||
|
|
||||||
| ## List correlations for a finding belonging to a log type | ||||||
|
|
||||||
| Used to list correlations for specific findings and the log types associated with them. | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Please rephrase as a complete sentence naming the noun (what is used to do this?). |
||||||
|
|
||||||
| ```json | ||||||
| GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m | ||||||
| ``` | ||||||
|
|
||||||
| ### Query parameters | ||||||
|
|
||||||
| | Parameter | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `finding` | String | The finding ID. | | ||||||
| | `detector_type` | String | The log type for the detector. | | ||||||
| | `nearby_findings` | Number | The number of nearby findings with respect to the given finding ID. | | ||||||
| | `time_window` | String | Sets a window of time in which all of the correlations must have occurred together. | | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
|
|
||||||
| #### Example request | ||||||
|
|
||||||
| ```json | ||||||
| GET /_plugins/_security_analytics/findings/correlate?finding=425dce0b-f5ee-4889-b0c0-7d15669f0871&detector_type=ad_ldap&nearby_findings=20&time_window=10m | ||||||
| ``` | ||||||
| {% include copy-curl.html %} | ||||||
|
|
||||||
| #### Example response | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "findings": [ | ||||||
| { | ||||||
| "finding": "5c661104-aaa9-484b-a91f-9cad4ae6d5f5", | ||||||
| "detector_type": "others_application", | ||||||
| "score": 0.000015182109564193524 | ||||||
| }, | ||||||
| { | ||||||
| "finding": "2485b623-6573-42f4-a055-9b927e38a65f", | ||||||
| "detector_type": "ad_ldap", | ||||||
| "score": 0.000001615897872397909 | ||||||
| }, | ||||||
| { | ||||||
| "finding": "051e00ad-5996-4c41-be20-f992451d1331", | ||||||
| "detector_type": "windows", | ||||||
| "score": 0.000016230604160227813 | ||||||
| }, | ||||||
| { | ||||||
| "finding": "f11ca8a3-50d7-4074-a951-51439aa9e67b", | ||||||
| "detector_type": "s3", | ||||||
| "score": 0.000001759401811796124 | ||||||
| }, | ||||||
| { | ||||||
| "finding": "9b86980e-5fb7-4c5a-bd1b-879a1e3baf12", | ||||||
| "detector_type": "network", | ||||||
| "score": 0.0000016306962606904563 | ||||||
| }, | ||||||
| { | ||||||
| "finding": "e7dea5a1-164f-48f9-880e-4ba33e508713", | ||||||
| "detector_type": "network", | ||||||
| "score": 0.00001632626481296029 | ||||||
| } | ||||||
| ] | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### Response fields | ||||||
|
|
||||||
| | Field | Type | Description | | ||||||
| | :--- | :--- |:--- | | ||||||
| | `finding` | String | The finding ID. | | ||||||
| | `detector_type` | String | The log type associated with the finding. | | ||||||
| | `score` | Number | The correlation score for the correlated finding. The score is based on the proximity of relevant findings in the threat scenario defined by the correlation rule. | | ||||||
|
|
||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,4 +18,5 @@ The APIs for Security Analytics are separated into the following categories: | |
| * [Rules APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/rule-api/) | ||
| * [Mappings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/) | ||
| * [Alerts and findings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/alert-finding-api/) | ||
| * [Correlation engine APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/correlation-eng/) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,17 @@ The following APIs can be used for a number of tasks related to mappings, from c | |
|
|
||
| ## Get Mappings View | ||
|
|
||
| Returns a view of the fields contained in an index used as a log source. | ||
|
There was a problem hiding this comment. Please rephrase as a complete sentence naming the noun (what is used to do this?). |
||
|
|
||
| ### Request fields | ||
|
|
||
| The following fields are used to get field mappings. | ||
|
|
||
| Field | Type | Description | ||
| :--- | :--- |:--- | | ||
| | `index_name` | String | The name of the index used for log ingestion. | | ||
| | `rule_topic` | String | The log type of the index. | | ||
|
|
||
| ### Example request | ||
|
|
||
| ```json | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear | |||||
|
|
||||||
| ## Create Custom Rule | ||||||
|
|
||||||
| The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma). | ||||||
cwillum marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Confirm capitalization of the API name. |
||||||
|
|
||||||
| ```json | ||||||
| POST /_plugins/_security_analytics/rules?category=windows | ||||||
|
|
||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -60,6 +60,15 @@ | |||||
|
|
||||||
| Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder. | ||||||
|
|
||||||
| #### Amazon Security Lake logs | ||||||
|
Check failure on line 63 in _security-analytics/sec-analytics-config/detectors-config.md
|
||||||
|
|
||||||
| [Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema). | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request. | ||||||
| {: .tip } | ||||||
|
|
||||||
| ### Automatically mapped fields | ||||||
|
|
||||||
| Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Expand **Automatically mapped fields** to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image. | ||||||
|
|
||||||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.