Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

4.6 destination flags #54

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

4.6 destination flags #54

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
62d80ea
Formatting fixes
eldarnash Jan 29, 2024
0c296d3
[4.6] Adds seqnum-all destination flag
eldarnash Feb 4, 2024
c9dd9bd
Fix review comments
eldarnash Feb 5, 2024
a43c7a5
threaded flag clarification
eldarnash Feb 5, 2024
34bd8f7
Includes flags option in a bunch of destinations where it was missing
eldarnash Feb 5, 2024
037189a
Formatting fix
eldarnash Feb 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions content/headless/chunk/option-destination-flags.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,31 @@

| | |
| -------- | ------------------------------ |
| Type: | no-multi-line, syslog-protocol |
| Type: | `ignore-errors`, `no-multi-line`, `no-seqnum-all`, `seqnum-all`, `syslog-protocol`, `threaded` |
| Default: | empty set |

*Description:* Flags influence the behavior of the destination driver.

- `ignore-errors`: <!-- FIXME -->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ignore-error flag silently drops messages which have not been successfully sent out because of unrecoverable low-level errors (e.g. I/O errors).

- `no-multi-line`: The `no-multi-line` flag disables line-breaking in the messages: the entire message is converted to a single line.
- `no-seqnum-all`: <!-- FIXME -->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The no-seqnum-all flag explicitly disables new seqnum generation for forwarded messages, which is the default behavior.

@bazsi Do I understand this right? I feel like this should return 0 seqnum even for locally generated messages, but the flag does not clear the LWO_SEQNUM field:
https://github.com/syslog-ng/syslog-ng/blob/1f0a854d0227dd95410fbc094997d8b0b225cd41/lib/logwriter.c#L2037

Do we even need this flag? The no-seqnum flag (not yet released) does the full disabling, that I have mentioned. I think it is confusing to have both.

Also, why do we need the seqnum flag, which sets the default behavior?

I feel like there are 3 cases:

  1. local messages get a generated seqnum, forwarded messages keep their own.
  2. all messages get a generated seqnum.
  3. neither local, nor forwarded messages get a seqnum (SEQNUM = 0).

The first is the default behavior, which does not necessarily need an explicit flag.
The second is enabled by seqnum-all.
The third is enabled by no-seqnum.

I don't see the point of having any other SEQNUM related flags.

- `seqnum-all`: Available in {{% param "product.abbrev" %}} version 4.6 and later. By default, {{% param "product.abbrev" %}} follows the logic of the RFC5424 `meta.sequenceId` structured data element: it adds a sequence number to local messages, forwarded messages retain their original sequenceId.

*Description:* Flags influence the behavior of the destination driver.
The `seqnum-all` flag adds a sequence number to every message sent to the destination, not just local messages. This also changes the behavior of the `${SEQNUM}` macro. For example:

```shell
destination { syslog("127.0.0.1" port(2001) flags(seqnum-all)); };
```

- *no-multi-line*: The `no-multi-line` flag disables line-breaking in the messages: the entire message is converted to a single line.
The output messages have increasing sequence numbers:

- *syslog-protocol*: The `syslog-protocol` flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the `syslog` driver, and that the `syslog` driver automatically adds the frame header to the messages.
```shell
<13>1 2023-12-09T21:51:30+00:00 localhost sdff - - [meta sequenceId="1"] f sdf fsd
<13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="2"] f sdf fsd
<13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="3"] f sdf fsd
<13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="4"] f sdf fsd
<13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="5"] f sdf fsd
```

- `syslog-protocol`: The `syslog-protocol` flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the `syslog` driver, and that the `syslog` driver automatically adds the frame header to the messages.
- `threaded`: <!-- FIXME -->
Copy link
Member

@alltilla alltilla Feb 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threaded flag enables multithreading for the current destination, even if the global threaded(no) option is set. For more information about multithreading, see: https://axoflow.com/docs/axosyslog-core/chapter-multithreading/configuring-multithreading/#example-enabling-multithreading

49 changes: 19 additions & 30 deletions content/headless/chunk/option-source-flags.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,26 @@

## flags()

| | |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| | |
| -------- | ----- |
| Type: | assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 |
| Default: | empty set |
| Default: | empty set |

*Description:* Specifies the log parsing options of the source.

- *assume-utf8*: The `assume-utf8` flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the `validate-utf8` flag.
- *assume-utf8*: The `assume-utf8` flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the `validate-utf8` flag.
- *empty-lines*: Use the `empty-lines` flag to keep the empty lines of the messages. By default, {{% param "product.abbrev" %}} removes empty lines automatically.
- *expect-hostname*: If the `expect-hostname` flag is enabled, {{% param "product.abbrev" %}} will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the `no-hostname` flag by default.
- *guess-timezone*: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
- *kernel*: The `kernel` flag makes the source default to the `LOG_KERN | LOG_NOTICE` priority if not specified otherwise.
- *no-header*: The `no-header` flag triggers {{% param "product.abbrev" %}} to parse only the `PRI` field of incoming messages, and put the rest of the message contents into `$MSG`.

- *empty-lines*: Use the `empty-lines` flag to keep the empty lines of the messages. By default, {{% param "product.abbrev" %}} removes empty lines automatically.

- *expect-hostname*: If the `expect-hostname` flag is enabled, {{% param "product.abbrev" %}} will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the `no-hostname` flag by default.

- *guess-timezone*: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.

- *kernel*: The `kernel` flag makes the source default to the `LOG_KERN | LOG_NOTICE` priority if not specified otherwise.

- *no-header*: The `no-header` flag triggers {{% param "product.abbrev" %}} to parse only the `PRI` field of incoming messages, and put the rest of the message contents into `$MSG`.

Its functionality is similar to that of the `no-parse` flag, except the `no-header` flag does not skip the `PRI` field.

{{% alert title="Note" color="info" %}}
Essentially, the `no-header` flag signals {{% param "product.abbrev" %}} that the `syslog` header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the `PRI` field) is put into `$MSG`.
{{% /alert %}}



### Example: using the no-header flag with the syslog-parser() parser

The following example illustrates using the `no-header` flag with the `syslog-parser()` parser:
Expand All @@ -41,9 +35,8 @@ Essentially, the `no-header` flag signals {{% param "product.abbrev" %}} that th
);
};
```


- *no-hostname*: Enable the `no-hostname` flag if the log message does not include the hostname of the sender host. That way {{% param "product.abbrev" %}} assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
- *no-hostname*: Enable the `no-hostname` flag if the log message does not include the hostname of the sender host. That way {{% param "product.abbrev" %}} assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

```shell
source s_dell {
Expand All @@ -54,21 +47,17 @@ Essentially, the `no-header` flag signals {{% param "product.abbrev" %}} that th
};
```

- *no-multi-line*: The `no-multi-line` flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the `file()` and `pipe()` drivers support multi-line messages.
- *no-multi-line*: The `no-multi-line` flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the `file()` and `pipe()` drivers support multi-line messages.
- *no-parse*: By default, {{% param "product.abbrev" %}} parses incoming messages as syslog messages. The `no-parse` flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The {{% param "product.abbrev" %}} application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the `${MESSAGE}` macro). This flag is useful for parsing messages not complying to the syslog format.

- *no-parse*: By default, {{% param "product.abbrev" %}} parses incoming messages as syslog messages. The `no-parse` flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The {{% param "product.abbrev" %}} application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the `${MESSAGE}` macro). This flag is useful for parsing messages not complying to the syslog format.

{{% include-headless "chunk/para-flags-no-parse.md" %}}

- *dont-store-legacy-msghdr*: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before `msg` in the following message: `Jan 22 10:06:11 host program:msg`). If you do not want to store the original header of the message, enable the `dont-store-legacy-msghdr` flag.
- *dont-store-legacy-msghdr*: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before `msg` in the following message: `Jan 22 10:06:11 host program:msg`). If you do not want to store the original header of the message, enable the `dont-store-legacy-msghdr` flag.

- *sanitize-utf8*: When using the `sanitize-utf8` flag, {{% param "product.abbrev" %}} converts non-UTF-8 input to an escaped form, which is valid UTF-8.
- *sanitize-utf8*: When using the `sanitize-utf8` flag, {{% param "product.abbrev" %}} converts non-UTF-8 input to an escaped form, which is valid UTF-8.
- *store-raw-message*: Save the original message as received from the client in the `${RAWMSG}` macro. You can forward this raw message in its original form to another AxoSyslog node using the [`syslog-ng()` destination]({{< relref "/chapter-destinations/destination-syslog-ng/_index.md" >}}), or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
- *syslog-protocol*: The `syslog-protocol` flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the `syslog` driver, which handles only messages that have a frame header.

- *store-raw-message*: Save the original message as received from the client in the `${RAWMSG}` macro. You can forward this raw message in its original form to another AxoSyslog node using the [`syslog-ng()` destination]({{< relref "/chapter-destinations/destination-syslog-ng/_index.md" >}}), or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
- *validate-utf8*: The `validate-utf8` flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see {{% xref "/chapter-concepts/concepts-message-structure/concepts-message-ietfsyslog/_index.md" %}}). If the BOM character is missing, but the message is otherwise UTF-8 compliant, {{% param "product_name" %}} automatically adds the BOM character to the message.

- *syslog-protocol*: The `syslog-protocol` flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the `syslog` driver, which handles only messages that have a frame header.

- *validate-utf8*: The `validate-utf8` flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see {{% xref "/chapter-concepts/concepts-message-structure/concepts-message-ietfsyslog/_index.md" %}}). If the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.

{{% include-headless "chunk/para-bom-definition.md" %}}

Loading