Skip to content

Latest commit

 

History

History
99 lines (70 loc) · 3.49 KB

Multiline.md

File metadata and controls

99 lines (70 loc) · 3.49 KB
Raw

Multiline Codec

The multiline codec processes multiple lines into a single event.

As long as lines match the specified pattern they are buffered. When a line is encountered that does not match, an event is flushed as dictated by the what option.

Example

- name: multiline
  patterns: 
  - "!^[0-9]{4}/[0-9]{2}/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} "
  what: previous
  previous timeout: 30s

JSON Reader

If the json reader is used, this codec will cause the loss of all fields except the message field, and any event that does not have a message field will be filtered and discarded. The output will be an event with a single field, message, containing the combined values of the matched lines according to the configuration.

Options

max multiline bytes

Number. Optional. Default: spool max bytes

The maximum multiline length to process. If a multiline block exeeds this length, it will be split across multiple events.

This setting can not be greater than the spool max bytes setting.

patterns

Array of Strings. Required

A list of regular expressions to match against each line.

These are applied in the order that they are specified. As soon as the required number of matches occurred (dictated by the match configuration that defaults to any), the pattern is considered matched, and the action specified by what takes place.

The pattern syntax is the RE2 Syntax.

To negate a pattern such that a line is considered to match when the pattern does not match, prefix the pattern with an exclamation mark ("!"). For example, the pattern "!^STARTEVENT" would match any line which does not start with the text, "STARTEVENT".

The pattern can also be prefixed with "=" to explicitly state that a pattern is not negated, which then allows a literal match of an exclamation mark at the start of the pattern. For example, "=!EVENT!" would match a line containing, "!EVENT!".

match

String. Optional. Default: "any"
Available values: "any", "all"

Specifies whether matching a single pattern must be matched or if all patterns must be matched.

previous timeout

Duration. Optional. Default: 0. Ignored when "what" != "previous"

When using previous, if previous timeout is not 0 any buffered lines will be flushed as a single event if no more lines are received within the specified time period.

what

String. Optional. Default: "previous"
Available values: "previous", "next"

  • previous: When the line matches, it belongs in the same event as the previous line. In other words, when matching stops treat the current line as the start of the next event. Flush the previously buffered lines as a single event and start a new buffer containing this line.
  • next: When the line matches, it belongs in the same event as the next line. In other words, when matching stops treat the current line as the end of the current event. Flush the previously buffered lines along with this line as a single event and start a new buffer.

A side effect of using previous is that an event will not be flushed until the first line of the next event is encountered. The previous timeout option offers a solution to this.