Skip to content

Commit

Permalink
Merge branch 'main' into filterx-draft
Browse files Browse the repository at this point in the history
  • Loading branch information
fekete-robert authored Sep 17, 2024
2 parents df81347 + 0d6ab54 commit e3402f0
Show file tree
Hide file tree
Showing 51 changed files with 247 additions and 365 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ To use this repository, you need the following installed locally:

- [npm](https://www.npmjs.com/)
- [Go](https://go.dev/)
- [Hugo (Extended version)](https://gohugo.io/), at least version 0.92.3
- [Hugo (Extended version)](https://gohugo.io/), at least version 0.92.3, but not higher than 0.122 (https://github.com/gohugoio/hugo/releases/tag/v0.122.0, see the **Assets** section for binaries)

1. Install the dependencies. Clone the repository and navigate to the directory:

Expand Down
12 changes: 10 additions & 2 deletions content/app-man-syslog-ng/dqtool.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ weight: 4900

{{% alert title="Note" color="info" %}}

The `dqtool` application is distributed with the {{% param "product.abbrev" %}} system logging application, and is usually part of the {{% param "product.abbrev" %}} package.
The `dqtool` application is distributed with the {{% param "product.abbrev" %}} system logging application, and is usually part of the {{% param "product.abbrev" %}} package.

{{% /alert %}}

Expand Down Expand Up @@ -71,13 +71,21 @@ Mar 3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta

<span id="dqtool-relocate"></span>

## The info command

`info [DISK-BUFFER_FILE]`

Shows status information about the specified disk-buffer file. For example:

{{< include-headless "chunk/dqtool-info-example.md" >}}

## The relocate command

`relocate [options] [files]`

Use the `relocate` command to move or rename disk-buffer (also called disk-queue) files. Note that this option modifies the persist file. Stop {{% param "product.abbrev" %}} before using this command.

The `cat` command has the following options:
The `relocate` command has the following options:

- `--all` or `-a`

Expand Down
2 changes: 1 addition & 1 deletion content/app-man-syslog-ng/syslog-ng-ctl.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ An example output:
global.payload_reallocs.stats.processed
global.msg_clones.stats.processed
global.sdata_updates.stats.processed
tag..source.s_tcp.stats.processed
tag..source.s_tcp.stats.processed
```

The `syslog-ng-ctl query list` command has the following options:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,22 @@ weight: 100

This section describes the format of a syslog message, according to the [legacy-syslog or BSD-syslog protocol](https://tools.ietf.org/search/rfc3164). A syslog message consists of the following parts:

- `[PRI]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-pri/_index.md" >}})`

- `[HEADER]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/_index.md" >}})`

- `[MSG]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/_index.md" >}})`
- [`PRI`]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-pri/_index.md" >}})
- [`HEADER`]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/_index.md" >}})
- [`MSG`]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/_index.md" >}})

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message:
The following is a sample syslog message

```shell
<133>Feb 25 14:09:07 webserver syslogd: restart
<133>Feb 25 14:09:07 webserver syslogd: restart
```

The message corresponds to the following format:

```shell
<priority>timestamp hostname application: message
<priority>timestamp hostname application: message
```

The different parts of the message are explained in the following sections.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ This section describes the `PRI` message part of a syslog message, according to

For further details about the `HEADER` and `MSG` parts of a syslog message, see the following sections:

- `[HEADER]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/_index.md" >}})`
- [`HEADER`]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-header/_index.md" >}})

- `[MSG]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/_index.md" >}})`
- [`MSG`]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-bsdsyslog/concepts-message-bsdsyslog-msg/_index.md" >}})


## The PRI message part
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,34 @@ weight: 300

This section describes the format of a syslog message, according to the [IETF-syslog protocol](https://tools.ietf.org/html/rfc5424). A syslog message consists of the following parts:

- `HEADER` (includes the `PRI` as well)

- `STRUCTURED-DATA`

- `MSG`
- `HEADER` (includes the `PRI` as well)
- `STRUCTURED-DATA`
- `MSG`

The following is a sample syslog message (source: https://tools.ietf.org/html/rfc5424):

```shell
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
```

The message corresponds to the following format:

```shell
<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG
```

- Facility is 4, severity is 2, so PRI is 34.

- The VERSION is 1.

- The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second.

- The message originated from a host that identifies itself as "mymachine.example.com".

- The APP-NAME is "su" and the PROCID is unknown.

- The MSGID is "ID47".

- The MSG is "'su root' failed for lonvick...", encoded in UTF-8.
- Facility is 4, severity is 2, so PRI is 34.
- The VERSION is 1.
- The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second.
- The message originated from a host that identifies itself as "mymachine.example.com".
- The APP-NAME is "su" and the PROCID is unknown.
- The MSGID is "ID47".
- The MSG is "'su root' failed for lonvick...", encoded in UTF-8.
- In this example, the encoding is defined by the BOM:

- In this example, the encoding is defined by the BOM:

{{% include-headless "chunk/para-bom-definition.md" %}}

- There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field.
- There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field.

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

Expand All @@ -57,80 +48,25 @@ Facility codes may slightly vary between different platforms. The AxoSyslog appl

{{% /alert %}}

syslog Message Facilities

Numerical Code


Facility

0

kernel messages

1

user-level messages

2

mail system

3

system daemons

4

security/authorization messages

5

messages generated internally by syslogd

6

line printer subsystem

7

network news subsystem

8

UUCP subsystem

9

clock daemon

10

security/authorization messages

11

FTP daemon

12

NTP subsystem

13

log audit

14

log alert

15

clock daemon

16-23

locally used facilities (local0-local7)
| Numerical Code | Facility |
| -------------- | ---------------------------------------- |
| 0 | kernel messages |
| 1 | user-level messages |
| 2 | mail system |
| 3 | system daemons |
| 4 | security/authorization messages |
| 5 | messages generated internally by syslogd |
| 6 | line printer subsystem |
| 7 | network news subsystem |
| 8 | UUCP subsystem |
| 9 | clock daemon |
| 10 | security/authorization messages |
| 11 | FTP daemon |
| 12 | NTP subsystem |
| 13 | log audit |
| 14 | log alert |
| 15 | clock daemon |
| 16-23 | locally used facilities (local0-local7) |

The following table lists the severity values.

Expand All @@ -152,17 +88,12 @@ syslog Message Severities

The HEADER part contains the following elements:

- *VERSION*: Version number of the syslog protocol standard. Currently this can only be `1`.

- *ISOTIMESTAMP*: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: `2006-06-13T15:58:00.123+01:00`.

- *HOSTNAME*: The machine that originally sent the message.

- *APPLICATION*: The device or application that generated the message

- *PID*: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

- *MESSAGEID*: The ID number of the message.
- *VERSION*: Version number of the syslog protocol standard. Currently this can only be `1`.
- *ISOTIMESTAMP*: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: `2006-06-13T15:58:00.123+01:00`.
- *HOSTNAME*: The machine that originally sent the message.
- *APPLICATION*: The device or application that generated the message
- *PID*: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.
- *MESSAGEID*: The ID number of the message.

{{% alert title="Note" color="info" %}}

Expand All @@ -172,29 +103,21 @@ The {{% param "product.abbrev" %}} application supports other timestamp formats

The {{% param "product.abbrev" %}} application will truncate the following fields:

- If *APP-NAME* is longer than 48 characters it will be truncated to 48 characters.

- If *PROC-ID* is longer than 128 characters it will be truncated to 128 characters.

- If *MSGID* is longer than 32 characters it will be truncated to 32 characters.

- If *HOSTNAME* is longer than 255 characters it will be truncated to 255 characters.


- If *APP-NAME* is longer than 48 characters it will be truncated to 48 characters.
- If *PROC-ID* is longer than 128 characters it will be truncated to 128 characters.
- If *MSGID* is longer than 32 characters it will be truncated to 32 characters.
- If *HOSTNAME* is longer than 255 characters it will be truncated to 255 characters.

## The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets (*[]*). Every block includes the ID of the block, and one or more *name=value* pairs. The AxoSyslog application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md" %}}). An example STRUCTURED-DATA block looks like:

```shell
[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]
```



## The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if the BOM

{{% include-headless "chunk/para-bom-definition.md" %}} character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.

Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ To send messages from {{% param "product.abbrev" %}} to HDFS, complete the follo

2. Download the Hadoop Distributed File System (HDFS) libraries (version 2.x) from <http://hadoop.apache.org/releases.html>.

3. Extract the HDFS libraries into a temporary directory, then collect the various `.jar` files into a single directory (for example, `/opt/hadoop/lib/`) where {{% param "product.abbrev" %}} can access them. You must specify this directory in the {{% param "product.abbrev" %}} configuration file. The files are located in the various `lib` directories under the `share/` directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are `common/hadoop-common-2.7.0.jar`, `common/libs/\*.jar`, `hdfs/hadoop-hdfs-2.7.0.jar`, `hdfs/lib/\*`, but this may change between Hadoop releases, so it is easier to copy every `.jar` file into a single directory.
3. Extract the HDFS libraries into a temporary directory, then collect the various `.jar` files into a single directory (for example, `/opt/hadoop/lib/`) where {{% param "product.abbrev" %}} can access them. You must specify this directory in the {{% param "product.abbrev" %}} configuration file. The files are located in the various `lib` directories under the `share/` directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are `common/hadoop-common-2.7.0.jar`, `common/libs/*.jar`, `hdfs/hadoop-hdfs-2.7.0.jar`, `hdfs/lib/*`, but this may change between Hadoop releases, so it is easier to copy every `.jar` file into a single directory.

Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ See {{% xref "/chapter-destinations/configuring-destinations-http-nonjava/plugin
|------+-----------------------------------+------------|
```

To customize the action to take for a particular response code, use the following format: `response-action(<response-code> => <action>`. To customize multiple response code-action pairs, separate them with a comma, for example:
To customize the action to take for a particular response code, use the arrow operator in the following format: `response-action(<response-code> => <action>`. To customize multiple response code-action pairs, separate them with a comma, for example:

```shell
http(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ For the `http` destination, use this option as `class-name("org.syslog_ng.http.H

{{% include-headless "chunk/option-destination-java-class-path.md" %}}

For the `http` destination, include the path to the java modules of {{% param "product.abbrev" %}}, for example, `class-path("/syslog-ng/install_dir/lib/syslog-ng/java-modules/\*.jar")`.
For the `http` destination, include the path to the java modules of {{% param "product.abbrev" %}}, for example, `class-path("/syslog-ng/install_dir/lib/syslog-ng/java-modules/*.jar")`.


{{< include-headless "chunk/option-destination-hook.md" >}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,9 @@ For more information about the default values of the `transaction.timeout.ms` Ka



{{% include-headless "chunk/option-destination-java-class-path.md" %}}
{{< include-headless "chunk/option-destination-java-class-path.md" >}}

For the `kafka` destination, include the path to the directory where you copied the required libraries (see {{% xref "/chapter-destinations/configuring-destinations-kafka/destination-kafka-prerequisites/_index.md" %}}), for example, `client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/\*.jar")`.
For the `kafka` destination, include the path to the directory where you copied the required libraries (see {{% xref "/chapter-destinations/configuring-destinations-kafka/destination-kafka-prerequisites/_index.md" %}}), for example, `client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/*.jar")`.

{{% alert title="Note" color="info" %}}

Expand All @@ -63,11 +63,11 @@ Unlike in the Java implementation, the `client-lib-dir()` option has no signific

## config()

*Description:* You can use this option to expand or override the options of the properties-file().
*Description:* You can use this option to expand or override the options of the `properties-file()`.

{{< include-headless "chunk/kafka-c-impl-required-options-note.md" >}}

The {{% param "product.abbrev" %}}`kafka` destination supports all properties of the official Kafka producer. For details, see [the librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).
The {{% param "product.abbrev" %}} `kafka` destination supports all properties of the official Kafka producer. For details, see [the librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).

The syntax of the config() option is the following:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following options are required: `kafka-bootstrap-servers()`, `topic()`. Note

{{% include-headless "chunk/option-destination-java-class-path.md" %}}

For the `kafka` destination, include the path to the directory where you copied the required libraries (see {{% xref "/chapter-destinations/configuring-destinations-kafka/destination-kafka-prerequisites/_index.md" %}}), for example, `client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/\*.jar")`.
For the `kafka` destination, include the path to the directory where you copied the required libraries (see {{% xref "/chapter-destinations/configuring-destinations-kafka/destination-kafka-prerequisites/_index.md" %}}), for example, `client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/*.jar")`.



Expand Down
2 changes: 1 addition & 1 deletion content/chapter-destinations/destination-loki/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ The `loki()` destination has the following options.

| | |
| -------- | ------- |
| Type: | |
| Type: | arrow list |
| Default: | See the description |

The labels applied to the message as they are sent to the destination. Use the following format:
Expand Down
2 changes: 1 addition & 1 deletion content/chapter-destinations/google-bigquery/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ Alternatively, you can set the schema with the [`schema()`](#schema) option.

| | |
| -------- | -------------------------- |
| Type: | See the description |
| Type: | arrow list |
| Default: | - |

*Description:* Sets the schema of the BigQuery table. On the left side of the arrow, set the name of the column and its type. On the right side, set any {{% param "product_name" %}} template or macro, which gets evaluated on each log that is routed to the `bigquery()` destination. The available column types are: `STRING`, `BYTES`, `INTEGER`, `FLOAT`, `BOOLEAN`, `TIMESTAMP`, `DATE`, `TIME`, `DATETIME`, `JSON`, `NUMERIC`, `BIGNUMERIC`, `GEOGRAPHY`, `RECORD`, `INTERVAL`. For example:
Expand Down
Loading

0 comments on commit e3402f0

Please sign in to comment.