Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Filterx docs #69

Merged
merged 89 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
3bf50c9
Filterx draft docs part 1
Aug 12, 2024
eb3e562
Regex matches, part 1
Aug 13, 2024
23e529d
Filterx function reference draft, part 1
Aug 13, 2024
734b382
Filterx function reference draft, part 2
Aug 13, 2024
200f4a9
strptime function docs
Aug 15, 2024
9fbf054
Adds stub for cache_json_file
Aug 15, 2024
74c5323
Adds flatten function
Aug 15, 2024
be028c5
Typpo fix
Aug 15, 2024
d4176c0
Adds unset_empties function
Aug 15, 2024
b45b080
Update content/filterx/_index.md
fekete-robert Aug 16, 2024
c3396c3
Update content/filterx/_index.md
fekete-robert Aug 16, 2024
2713ab7
Update content/filterx/filterx-boolean/_index.md
fekete-robert Aug 16, 2024
329b03b
Update content/filterx/filterx-comparing/_index.md
fekete-robert Aug 16, 2024
b52fe9b
Review comment fixes
Aug 16, 2024
854fca8
Formatting fixes
Aug 16, 2024
598d1eb
Formatting fixes
Aug 16, 2024
ef66920
Review comment fixes
Aug 16, 2024
d9a169f
cache_json_file update
Aug 16, 2024
f7503e6
regexp updates
Aug 21, 2024
aa9e496
Fixes from furiel's review
Aug 26, 2024
134a8ba
Typo fix
Aug 26, 2024
49afa2e
regexp_search filterx function
Aug 26, 2024
eefd055
regexp_subst docs
Aug 26, 2024
27dbc0a
Formatting fixes
Aug 26, 2024
e209a4c
Adds parse_kv filterx parser
Aug 26, 2024
6e8aa0c
Typo fixes and minor source reorg in legacy csv-parser
Aug 27, 2024
12300b4
parse_csv filterx draft docs
Aug 27, 2024
5f76d47
Minor fixes
Aug 27, 2024
e185800
Small fixes
Aug 27, 2024
c5978a3
Unset accepts multiple arguments
Aug 27, 2024
e137bf2
Clarifications in len
Aug 27, 2024
5e0e55a
Adds format_kv
Aug 27, 2024
ba2744b
format_csv updates
Aug 27, 2024
7bd175f
Clarifications and link fixes
Aug 27, 2024
aa6b98b
Adds the += operator
Aug 27, 2024
bb0df7d
Minor cleanups
Aug 28, 2024
4e31729
Clarifications
Aug 28, 2024
609d05f
Mention case insensitive flag for regexes
Aug 28, 2024
9aa0dfa
Formatting and typo fix
Aug 28, 2024
4c942a7
Adds some examples based on old rewrite usecases
Aug 28, 2024
0fa1b3d
Adds updating filters section
Aug 29, 2024
97b0cf3
Minor fixes
Aug 29, 2024
2ea8794
More on updating old blocks
Aug 29, 2024
9f4514c
Adds draft use case example
Aug 30, 2024
9e8e849
Review comment fixes
Sep 2, 2024
0191133
Formatting fixes and clarifications
Sep 2, 2024
592f2d8
Adds experiental banner to filterx pages
Sep 3, 2024
1c05872
Use curly braces more consistently
Sep 3, 2024
4cd68d9
Adds rough draft for handling OTEL logs
Sep 3, 2024
d7d9be9
String and template function clarifications
Sep 4, 2024
6f917f8
Minor fixes
Sep 4, 2024
0bf5ea3
Small fixes
Sep 4, 2024
cea0641
Otel example fix
Sep 4, 2024
2af7fbe
Hide parse_xml placeholder
Sep 4, 2024
227b51d
more otel example fixes
Sep 4, 2024
67f5329
Draft otel structures reference
Sep 4, 2024
ab15fa1
Hide draft link
Sep 4, 2024
7576947
syslog to otel mapping example updates
Sep 5, 2024
575f63b
OTel routing example
Sep 5, 2024
aacdc34
Minor fixes
Sep 5, 2024
4a424bf
Moves updating filters section to separate page
Sep 5, 2024
ee76f19
Title fix
Sep 5, 2024
7b1af5b
Remove mention of $0 match group
Sep 6, 2024
efecefd
Minor updates
Sep 6, 2024
ecbe4f9
Delete unneeded comments
Sep 9, 2024
69634a5
Consistency improvements
Sep 9, 2024
e1095bc
Docs for the strict inequality operator
Sep 9, 2024
cc594b8
Link to strict inequality operator
Sep 9, 2024
a62b4d6
Typo fixes
Sep 10, 2024
56d0612
More json example
Sep 10, 2024
c128721
Adds missing id
Sep 10, 2024
7f1925f
Reorder subpages
Sep 10, 2024
2b141ac
Minor fixes
Sep 10, 2024
4f08423
Typo fix
Sep 10, 2024
4b8e8e6
Fixes from alltilla's review, part 1
Sep 10, 2024
e498cef
Updates for alltila's boolean review
Sep 11, 2024
83e47d4
Moves reusable filterx blocks to separate page and other fixes
Sep 11, 2024
93fcff6
More review fixes
Sep 11, 2024
4c47ae6
List/json clarifications
Sep 12, 2024
af14632
Small fixes
Sep 12, 2024
7cabb8d
Otel review fixes part 1
Sep 12, 2024
bff3737
Otel review fixes part 2
Sep 12, 2024
75e743e
Comparison fixes
Sep 12, 2024
53fc0a3
Mention filterx variables vs destinations
Sep 12, 2024
0caeb1f
Review fixes
Sep 13, 2024
9664c47
More review fixes
Sep 13, 2024
df81347
Fix datetime units
Sep 16, 2024
e3402f0
Merge branch 'main' into filterx-draft
fekete-robert Sep 17, 2024
68fcca1
Fix csv-parser default value
Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,48 +8,43 @@ In {{% param "product.abbrev" %}} version 3.8 and later, you can use an external

The database file is a simple text file in comma-separated value (CSV) format, where each line contains the following information:

- A selector or ID that appears in the log messages, for example, the hostname. To use shell-style globbing (wildcards) in selectors, see {{% xref "/chapter-enrich-data/data-enrichment-add-contextual-data/add-contextual-data-globs/_index.md" %}}. You can also reference the name of a filter that matches the messages, see {{% xref "/chapter-enrich-data/data-enrichment-add-contextual-data/add-contextual-data-filters/_index.md" %}}

- The name of the name-value pair that {{% param "product.abbrev" %}} adds to matching log messages.

- The value of the name-value pairs. Starting with {{% param "product.abbrev" %}} version 3.22, the value of the name-value pair can be a template or a template function, for example, `"selector3,name,$(echo $HOST_FROM)";`
- A selector or ID that appears in the log messages, for example, the hostname. To use shell-style globbing (wildcards) in selectors, see {{% xref "/chapter-enrich-data/data-enrichment-add-contextual-data/add-contextual-data-globs/_index.md" %}}. You can also reference the name of a filter that matches the messages, see {{% xref "/chapter-enrich-data/data-enrichment-add-contextual-data/add-contextual-data-filters/_index.md" %}}
- The name of the name-value pair that {{% param "product.abbrev" %}} adds to matching log messages.
- The value of the name-value pairs. Starting with {{% param "product.abbrev" %}} version 3.22, the value of the name-value pair can be a template or a template function, for example, `"selector3,name,$(echo $HOST_FROM)";`

For example, the following csv-file contains three lines identified with the IP address, and adds the `host-role` field to the log message.

```shell
192.168.1.1,host-role,webserver
192.168.2.1,host-role,firewall
192.168.3.1,host-role,mailserver
192.168.1.1,host-role,webserver
192.168.2.1,host-role,firewall
192.168.3.1,host-role,mailserver
```


## The database file: {#add-contextual-data-csv-database-file}

The database file must comply with the [RFC4180 CSV format](https://tools.ietf.org/html/rfc4180), with the following exceptions and limitations:

- The values of the CSV-file cannot contain line-breaks
- The values of the CSV-file cannot contain line-breaks

To add multiple name-value pairs to a message, include a separate line in the database for each name-value pair, for example:

```shell
192.168.1.1,host-role,webserver
192.168.1.1,contact-person,"John Doe"
192.168.1.1,contact-email,[email protected]
192.168.1.1,host-role,webserver
192.168.1.1,contact-person,"John Doe"
192.168.1.1,contact-email,[email protected]
```

Technically, `add-contextual-data()` is a parser in {{% param "product.abbrev" %}} so you have to define it as a parser object.



## Declaration:

```shell
parser p_add_context_data {
add-contextual-data(
selector("${HOST}"),
database("context-info-db.csv"),
);
};
parser p_add_context_data {
add-contextual-data(
selector("${HOST}"),
database("context-info-db.csv"),
);
};
```


Expand All @@ -63,31 +58,30 @@ If you modify the database file, you have to reload {{% param "product.abbrev" %
The following example defines uses a CSV database to add the role of the host based on its IP address, and prefixes the added name-value pairs with `.metadata`. The destination includes a template that simply appends the added name-value pairs to the end of the log message.

```shell
@include "scl.conf"
source s_network {
network(port(5555));
};
destination d_local {
file("/tmp/test-msgs.log"
template("$MSG Additional metadata:[${.metadata.host-role}]")};
parser p_add_context_data {
add-contextual-data(selector("$SOURCEIP"), database("context-info-db.csv"), default-selector("unknown"), prefix(".metadata."));
};
log {
source(s_network);
parser(p_add_context_data);
destination(d_local);
};
@include "scl.conf"

source s_network {
network(port(5555));
};

destination d_local {
file("/tmp/test-msgs.log"
template("$MSG Additional metadata:[${.metadata.host-role}]")};

parser p_add_context_data {
add-contextual-data(selector("$SOURCEIP"), database("context-info-db.csv"), default-selector("unknown"), prefix(".metadata."));
};

log {
source(s_network);
parser(p_add_context_data);
destination(d_local);
};
```

```shell
192.168.1.1,host-role,webserver
192.168.2.1,host-role,firewall
192.168.3.1,host-role,mailserver
unknown,host-role,unknown
192.168.1.1,host-role,webserver
192.168.2.1,host-role,firewall
192.168.3.1,host-role,mailserver
unknown,host-role,unknown
```

Original file line number Diff line number Diff line change
Expand Up @@ -16,60 +16,7 @@ The `date-parser()` parser has the following options.

*Description:* Specifies the format how {{% param "product.abbrev" %}} should parse the date. You can use the following format elements:

```shell
%% PERCENT
%a day of the week, abbreviated
%A day of the week
%b month abbr
%B month
%c MM/DD/YY HH:MM:SS
%C ctime format: Sat Nov 19 21:05:57 1994
%d numeric day of the month, with leading zeros (eg 01..31)
%e like %d, but a leading zero is replaced by a space (eg 1..31)
%f microseconds, leading 0's, extra digits are silently discarded
%D MM/DD/YY
%G GPS week number (weeks since January 6, 1980)
%h month, abbreviated
%H hour, 24 hour clock, leading 0's)
%I hour, 12 hour clock, leading 0's)
%j day of the year
%k hour
%l hour, 12 hour clock
%L month number, starting with 1
%m month number, starting with 01
%M minute, leading 0's
%n NEWLINE
%o ornate day of month -- "1st", "2nd", "25th", etc.
%p AM or PM
%P am or pm (Yes %p and %P are backwards :)
%q Quarter number, starting with 1
%r time format: 09:05:57 PM
%R time format: 21:05
%s seconds since the Epoch, UCT
%S seconds, leading 0's
%t TAB
%T time format: 21:05:57
%U week number, Sunday as first day of week
%w day of the week, numerically, Sunday == 0
%W week number, Monday as first day of week
%x date format: 11/19/94
%X time format: 21:05:57
%y year (2 digits)
%Y year (4 digits)
%Z timezone in ascii format (for example, PST), or in format -/+0000
%z timezone in ascii format (for example, PST), or in format -/+0000 (Required element)
```

{{% alert title="Warning" color="warning" %}}

When using the %z and %Z format elements, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is not specified, %Z does not trigger a warning if the timezone is not specified.

For further information about the %z and %Z format elements, see the 'DESCRIPTION' section on the [srtptime(3) - NetBSD Manual Pages](https://man.netbsd.org/NetBSD-7.0/i386/strptime.3).

{{% /alert %}}


For example, for the date `01/Jan/2016:13:05:05 PST` use the following format string: `format("%d/%b/%Y:%H:%M:%S %Z")`
{{< include-headless "chunk/date-string-format.md" >}}

{{% include-headless "chunk/example-date-parser.md" %}}

Expand Down
Loading
Loading