[helm]: implement system integration as chart built-in #5855
+788
−19
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Example: System Integration with custom auth log paths | ||
|
|
||
| In this example we install the built-in `system` integration and specify custom paths for the auth logs stream (see [agent-system-values.yaml](agent-system-values.yaml)). | ||
|
|
||
| ## Prerequisites: | ||
| 1. A k8s secret that contains the connection details to an Elasticsearch cluster such as the URL and the API key ([Kibana - Creating API Keys](https://www.elastic.co/guide/en/kibana/current/api-keys.html)): | ||
| ```console | ||
| kubectl create secret generic es-api-secret \ | ||
| --from-literal=api_key=... \ | ||
| --from-literal=url=... | ||
| ``` | ||
|
|
||
| 2. `system` integration assets installed through Kibana ([Kibana - Install and uninstall Elastic Agent integration assets](https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html)) | ||
|
|
||
| ## Run: | ||
| ```console | ||
| helm install elastic-agent ../../ \ | ||
| -f ./agent-system-values.yaml \ | ||
| --set outputs.default.type=ESSecretAuthAPI \ | ||
| --set outputs.default.secretName=es-api-secret | ||
| ``` | ||
|
|
||
| ## Validate: | ||
|
|
||
| 1. The Kibana `system`-related dashboards should start showing up the respective info. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| system: | ||
|
There was a problem hiding this comment. We should have rendered examples for this, right? I thought the mage target would error in this case. There was a problem hiding this comment. hmm, I thought I added it but this never happened, but the CI should fail yes There was a problem hiding this comment. oh I suspect that this is because the file was never versioned and the There was a problem hiding this comment. Shouldn't it fail anyway if it can't find the file? I didn't know the examples mage target was aware of versioning. Anyway, this isn't a blocker for this PR, but we should make sure the check actually catches issues. There was a problem hiding this comment. the current CI checks flow is the following:
100% agreed this needs to be extended |
||
| enabled: true | ||
| authLogs: | ||
| vars: | ||
| paths: | ||
| - /var/log/custom_auth.log | ||
|
|
||
| agent: | ||
| unprivileged: true | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| {{- define "elasticagent.system.init" -}} | ||
| {{- if eq $.Values.system.enabled true -}} | ||
| {{- include "elasticagent.system.config.logs.init" $ -}} | ||
| {{- include "elasticagent.system.config.metrics.init" $ -}} | ||
| {{- end -}} | ||
| {{- end -}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| {{- define "elasticagent.system.config.logs.init" -}} | ||
| {{- if eq $.Values.system.enabled true }} | ||
| {{- $preset := $.Values.agent.presets.perNode -}} | ||
| {{- $inputVal := (include "elasticagent.system.config.logs.input" $ | fromYaml) -}} | ||
| {{- if ($inputVal).streams }} | ||
| {{- include "elasticagent.preset.mutate.inputs" (list $ $preset (list $inputVal)) -}} | ||
| {{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}} | ||
| {{- end -}} | ||
| {{- end -}} | ||
| {{- end -}} | ||
|
|
||
| {{- define "elasticagent.system.config.logs.input" -}} | ||
| id: system-logs | ||
| type: logfile | ||
| use_output: {{ $.Values.system.output }} | ||
| data_stream: | ||
| namespace: {{ $.Values.system.namespace }} | ||
| streams: | ||
| {{- if eq $.Values.system.authLogs.enabled true }} | ||
| - data_stream: | ||
| dataset: system.auth | ||
| type: logs | ||
| multiline: | ||
| pattern: ^\s | ||
| match: after | ||
| {{- $vars := (include "elasticagent.system.config.auth_logs.default_vars" .) | fromYaml -}} | ||
| {{- mergeOverwrite $vars $.Values.system.authLogs.vars | toYaml | nindent 4 }} | ||
| {{- end }} | ||
| {{- if eq $.Values.system.syslog.enabled true }} | ||
| - data_stream: | ||
| dataset: system.syslog | ||
| type: logs | ||
| multiline: | ||
| pattern: ^\s | ||
| match: after | ||
| {{- $vars := (include "elasticagent.system.config.syslog.default_vars" .) | fromYaml -}} | ||
| {{- mergeOverwrite $vars $.Values.system.syslog.vars | toYaml | nindent 4 }} | ||
| {{- end }} | ||
| {{- end -}} | ||
|
|
||
| {{/* | ||
| Defaults for auth log input stream | ||
| */}} | ||
| {{- define "elasticagent.system.config.auth_logs.default_vars" -}} | ||
| paths: | ||
| - /var/log/auth.log* | ||
| - /var/log/secure* | ||
| exclude_files: | ||
| - \.gz$ | ||
| processors: | ||
| - add_locale: null | ||
| tags: | ||
| - system-auth | ||
| ignore_older: 72h | ||
| {{- end -}} | ||
|
|
||
| {{/* | ||
| Defaults for auth log syslog stream | ||
| */}} | ||
| {{- define "elasticagent.system.config.syslog.default_vars" -}} | ||
| paths: | ||
| - /var/log/messages* | ||
| - /var/log/syslog* | ||
| - /var/log/system* | ||
| exclude_files: | ||
| - \.gz$ | ||
| processors: | ||
| - add_locale: null | ||
| tags: null | ||
| ignore_older: 72h | ||
| {{- end -}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| {{- define "elasticagent.system.config.metrics.init" -}} | ||
| {{- if $.Values.system.metrics.enabled}} | ||
| {{- $preset := $.Values.agent.presets.perNode -}} | ||
| {{- $inputVal := (include "elasticagent.system.config.metrics.input" $ | fromYamlArray) -}} | ||
| {{- include "elasticagent.preset.mutate.inputs" (list $ $preset $inputVal) -}} | ||
| {{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}} | ||
| {{- end -}} | ||
| {{- end -}} | ||
|
|
||
| {{- define "elasticagent.system.config.metrics.input" -}} | ||
| - id: system-metrics | ||
| type: system/metrics | ||
| use_output: {{ $.Values.system.output }} | ||
| data_stream: | ||
| namespace: {{ $.Values.system.namespace }} | ||
| streams: | ||
| - data_stream: | ||
| dataset: system.cpu | ||
| type: metrics | ||
| period: 10s | ||
| cpu.metrics: | ||
| - percentages | ||
| - normalized_percentages | ||
| metricsets: | ||
| - cpu | ||
| - data_stream: | ||
| dataset: system.diskio | ||
| type: metrics | ||
| period: 10s | ||
| diskio.include_devices: null | ||
|
There was a problem hiding this comment. Are these nulls intentional? I would expect this to either be an empty list or for the key to not exist at all. There was a problem hiding this comment. this crossed my mind as well, but then I wondered why this config was rendered like that from Kibana; thus, I figured not to mess with it 😄 but sure this can be further optimised in the future |
||
| metricsets: | ||
| - diskio | ||
| - data_stream: | ||
| dataset: system.filesystem | ||
| type: metrics | ||
| period: 1m | ||
| metricsets: | ||
| - filesystem | ||
| processors: | ||
| - drop_event.when.regexp: | ||
| system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/) | ||
| - data_stream: | ||
| dataset: system.fsstat | ||
| type: metrics | ||
| period: 1m | ||
| metricsets: | ||
| - fsstat | ||
| processors: | ||
| - drop_event.when.regexp: | ||
| system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/) | ||
| - data_stream: | ||
| dataset: system.load | ||
| type: metrics | ||
| condition: '${host.platform} != ''windows''' | ||
| period: 10s | ||
| metricsets: | ||
| - load | ||
| - data_stream: | ||
| dataset: system.memory | ||
| type: metrics | ||
| period: 10s | ||
| metricsets: | ||
| - memory | ||
| - data_stream: | ||
| dataset: system.network | ||
| type: metrics | ||
| period: 10s | ||
| network.interfaces: null | ||
| metricsets: | ||
| - network | ||
| - data_stream: | ||
| dataset: system.process | ||
| type: metrics | ||
| period: 10s | ||
| processes: | ||
| - .* | ||
| process.include_top_n.by_cpu: 5 | ||
| process.include_top_n.by_memory: 5 | ||
| process.cmdline.cache.enabled: true | ||
| process.cgroups.enabled: false | ||
| process.include_cpu_ticks: false | ||
| metricsets: | ||
| - process | ||
| process.include_cpu_ticks: false | ||
| - data_stream: | ||
| dataset: system.process_summary | ||
| type: metrics | ||
| period: 10s | ||
| metricsets: | ||
| - process_summary | ||
| - data_stream: | ||
| dataset: system.socket_summary | ||
| type: metrics | ||
| period: 10s | ||
| metricsets: | ||
| - socket_summary | ||
| - data_stream: | ||
| type: metrics | ||
| dataset: system.uptime | ||
| metricsets: | ||
| - uptime | ||
| period: 10s | ||
| {{- end -}} | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the default
false? I feel like most will want this on, it brings immediate value to the Observability section in Kibana.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More than happy to have it by default as
true, it also makes sense to me. @nimarezainia what do you think on that? 🙂