-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[helm]: implement system integration as chart built-in #5855
Changes from 3 commits
3dd64de
b0239d6
70205a8
da31f5e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
# Example: System Integration with custom auth log paths | ||
|
||
In this example we install the built-in `system` integration and specify custom paths for the auth logs stream (see [agent-system-values.yaml](agent-system-values.yaml)). | ||
|
||
## Prerequisites: | ||
1. A k8s secret that contains the connection details to an Elasticsearch cluster such as the URL and the API key ([Kibana - Creating API Keys](https://www.elastic.co/guide/en/kibana/current/api-keys.html)): | ||
```console | ||
kubectl create secret generic es-api-secret \ | ||
--from-literal=api_key=... \ | ||
--from-literal=url=... | ||
``` | ||
|
||
2. `system` integration assets installed through Kibana ([Kibana - Install and uninstall Elastic Agent integration assets](https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html)) | ||
|
||
## Run: | ||
```console | ||
helm install elastic-agent ../../ \ | ||
-f ./agent-system-values.yaml \ | ||
--set outputs.default.type=ESSecretAuthAPI \ | ||
--set outputs.default.secretName=es-api-secret | ||
``` | ||
|
||
## Validate: | ||
|
||
1. The Kibana `system`-related dashboards should start showing up the respective info. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
system: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should have rendered examples for this, right? I thought the mage target would error in this case. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. hmm, I thought I added it but this never happened, but the CI should fail yes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. oh I suspect that this is because the file was never versioned and the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shouldn't it fail anyway if it can't find the file? I didn't know the examples mage target was aware of versioning. Anyway, this isn't a blocker for this PR, but we should make sure the check actually catches issues. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the current CI checks flow is the following:
100% agreed this needs to be extended |
||
enabled: true | ||
authLogs: | ||
vars: | ||
paths: | ||
- /var/log/custom_auth.log | ||
|
||
agent: | ||
unprivileged: true |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{{- define "elasticagent.system.init" -}} | ||
{{- if eq $.Values.system.enabled true -}} | ||
{{- include "elasticagent.system.config.logs.init" $ -}} | ||
{{- include "elasticagent.system.config.metrics.init" $ -}} | ||
{{- end -}} | ||
{{- end -}} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
{{- define "elasticagent.system.config.logs.init" -}} | ||
{{- if eq $.Values.system.enabled true }} | ||
{{- $preset := $.Values.agent.presets.perNode -}} | ||
{{- $inputVal := (include "elasticagent.system.config.logs.input" $ | fromYaml) -}} | ||
{{- if ($inputVal).streams }} | ||
{{- include "elasticagent.preset.mutate.inputs" (list $ $preset (list $inputVal)) -}} | ||
{{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}} | ||
{{- end -}} | ||
{{- end -}} | ||
{{- end -}} | ||
|
||
{{- define "elasticagent.system.config.logs.input" -}} | ||
id: system-logs | ||
type: logfile | ||
use_output: {{ $.Values.system.output }} | ||
data_stream: | ||
namespace: {{ $.Values.system.namespace }} | ||
streams: | ||
{{- if eq $.Values.system.authLogs.enabled true }} | ||
- data_stream: | ||
dataset: system.auth | ||
type: logs | ||
multiline: | ||
pattern: ^\s | ||
match: after | ||
{{- $vars := (include "elasticagent.system.config.auth_logs.default_vars" .) | fromYaml -}} | ||
{{- mergeOverwrite $vars $.Values.system.authLogs.vars | toYaml | nindent 4 }} | ||
{{- end }} | ||
{{- if eq $.Values.system.syslog.enabled true }} | ||
- data_stream: | ||
dataset: system.syslog | ||
type: logs | ||
multiline: | ||
pattern: ^\s | ||
match: after | ||
{{- $vars := (include "elasticagent.system.config.syslog.default_vars" .) | fromYaml -}} | ||
{{- mergeOverwrite $vars $.Values.system.syslog.vars | toYaml | nindent 4 }} | ||
{{- end }} | ||
{{- end -}} | ||
|
||
{{/* | ||
Defaults for auth log input stream | ||
*/}} | ||
{{- define "elasticagent.system.config.auth_logs.default_vars" -}} | ||
paths: | ||
- /var/log/auth.log* | ||
- /var/log/secure* | ||
exclude_files: | ||
- \.gz$ | ||
processors: | ||
- add_locale: null | ||
tags: | ||
- system-auth | ||
ignore_older: 72h | ||
{{- end -}} | ||
|
||
{{/* | ||
Defaults for auth log syslog stream | ||
*/}} | ||
{{- define "elasticagent.system.config.syslog.default_vars" -}} | ||
paths: | ||
- /var/log/messages* | ||
- /var/log/syslog* | ||
- /var/log/system* | ||
exclude_files: | ||
- \.gz$ | ||
processors: | ||
- add_locale: null | ||
tags: null | ||
ignore_older: 72h | ||
{{- end -}} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
{{- define "elasticagent.system.config.metrics.init" -}} | ||
{{- if $.Values.system.metrics.enabled}} | ||
{{- $preset := $.Values.agent.presets.perNode -}} | ||
{{- $inputVal := (include "elasticagent.system.config.metrics.input" $ | fromYamlArray) -}} | ||
{{- include "elasticagent.preset.mutate.inputs" (list $ $preset $inputVal) -}} | ||
{{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}} | ||
{{- end -}} | ||
{{- end -}} | ||
|
||
{{- define "elasticagent.system.config.metrics.input" -}} | ||
- id: system-metrics | ||
type: system/metrics | ||
use_output: {{ $.Values.system.output }} | ||
data_stream: | ||
namespace: {{ $.Values.system.namespace }} | ||
streams: | ||
- data_stream: | ||
dataset: system.cpu | ||
type: metrics | ||
period: 10s | ||
cpu.metrics: | ||
- percentages | ||
- normalized_percentages | ||
metricsets: | ||
- cpu | ||
- data_stream: | ||
dataset: system.diskio | ||
type: metrics | ||
period: 10s | ||
diskio.include_devices: null | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are these nulls intentional? I would expect this to either be an empty list or for the key to not exist at all. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this crossed my mind as well, but then I wondered why this config was rendered like that from Kibana; thus, I figured not to mess with it 😄 but sure this can be further optimised in the future |
||
metricsets: | ||
- diskio | ||
- data_stream: | ||
dataset: system.filesystem | ||
type: metrics | ||
period: 1m | ||
metricsets: | ||
- filesystem | ||
processors: | ||
- drop_event.when.regexp: | ||
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/) | ||
- data_stream: | ||
dataset: system.fsstat | ||
type: metrics | ||
period: 1m | ||
metricsets: | ||
- fsstat | ||
processors: | ||
- drop_event.when.regexp: | ||
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/) | ||
- data_stream: | ||
dataset: system.load | ||
type: metrics | ||
condition: '${host.platform} != ''windows''' | ||
period: 10s | ||
metricsets: | ||
- load | ||
- data_stream: | ||
dataset: system.memory | ||
type: metrics | ||
period: 10s | ||
metricsets: | ||
- memory | ||
- data_stream: | ||
dataset: system.network | ||
type: metrics | ||
period: 10s | ||
network.interfaces: null | ||
metricsets: | ||
- network | ||
- data_stream: | ||
dataset: system.process | ||
type: metrics | ||
period: 10s | ||
processes: | ||
- .* | ||
process.include_top_n.by_cpu: 5 | ||
process.include_top_n.by_memory: 5 | ||
process.cmdline.cache.enabled: true | ||
process.cgroups.enabled: false | ||
process.include_cpu_ticks: false | ||
metricsets: | ||
- process | ||
process.include_cpu_ticks: false | ||
- data_stream: | ||
dataset: system.process_summary | ||
type: metrics | ||
period: 10s | ||
metricsets: | ||
- process_summary | ||
- data_stream: | ||
dataset: system.socket_summary | ||
type: metrics | ||
period: 10s | ||
metricsets: | ||
- socket_summary | ||
- data_stream: | ||
type: metrics | ||
dataset: system.uptime | ||
metricsets: | ||
- uptime | ||
period: 10s | ||
{{- end -}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the default
false
? I feel like most will want this on, it brings immediate value to the Observability section in Kibana.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More than happy to have it by default as
true
, it also makes sense to me. @nimarezainia what do you think on that? 🙂