Add LogPatternTool #413

qianheng-aws · 2024-09-26T11:52:52Z

Description

Add LogPatternTool to support generate log patterns based on an input DSL.

Background of log pattern

Log pattern is a feature in Observability. It aims to find the common feature among retrieved logs.

The implementation of the log pattern feature in Observability includes 3 steps:

execute a PPL to retrieve the first log
set default pattern field, which is implemented in front-end. The default pattern field is found by scanning over the values of the first sample log. The field with the longest length of value will be the default pattern field. (Code Ref; For alert analysis, it seems meaningless to find pattern on the field with the longest length of value.)
execute a PPL to retrieve data and fill in log pattern table. Assuming field message is the pattern field, an example of such an PPL is:
source=opensearch_dashboards_sample_data_flights | where timestamp >= ’[2024-09-09 16](tel:2024090916):16:30.321000' and timestamp <= '2024-09-10 07:16:30.322000' | patterns Dest | stats count(), take(Dest, 1) by patterns_field
By default, the operator pattern in PPL will generate a new field named pattern_field which will ignore characters conatined by [a-zA-Z\d] in the original value(Code Ref; For alert analysis, it also seems less helpful to analysis on such pattern).

However, if customers use DSL to retrieve logs, it doesn't support such feature. Thus we need this tool to achieve this goal.

Implementation of LogPatternTool

This tool supports generating log patterns on the input dsl and index. It's implemented by
several steps:

Retrival [[${DOC_SIZE_FIELD}]] logs from index
Extract patterns for each retrieved log
2.1 Find Pattern Field: If users provide parameter [[${PATTERN_FIELD}]], use it as the pattern
field; Otherwise, find the string field with the longest length on the first log.
2.2 Extract Pattern: If users provide parameter [[${PATTERN}]], compile it as a pattern;
Otherwise, use [[${DEFAULT_IGNORED_CHARS}]]. It will remove all chars matching the pattern.
Group logs by their extracted patterns.
Find top N patterns with the largest sample log size.
For each found top N patterns, return [[${SAMPLE_LOG_SIZE}]] sample logs.

Example

POST _plugins/_ml/agents/V4pYKJIBf_90oTl_gmnp/_execute
{
    "parameters": {
       "selected_tools": ["LogPatternTool"],
       "top_n_pattern": "3",
       "sample_log_size": "20",
       "doc_size": 100,
        "input":  """
{"query":{"bool":{"filter":[{"range":{"timestamp":{"from":"1727172421713||-1h","to":"1727172421713","include_lower":true,"include_upper":true,"boost":1}}},{"term":{"response":{"value":"200","boost":1}}}],"adjust_pure_negative":true,"boost":1}}}
        """
    }
}

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.