Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[helm]: implement system integration as chart built-in #5855

Merged
merged 4 commits into from
Oct 30, 2024
+788 −19
Merged

[helm]: implement system integration as chart built-in #5855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3dd64de
feat: implement system integration as chart built-in
pkoutsovasilis Oct 25, 2024
b0239d6
feat: restructure logs streams in system integration to allow passing…
pkoutsovasilis Oct 30, 2024
70205a8
doc: add an example of system integration
pkoutsovasilis Oct 30, 2024
da31f5e
feat: add missing rendered file for system integration example
pkoutsovasilis Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions deploy/helm/elastic-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,12 +123,24 @@ The chart built-in [kubernetes integration](https://docs.elastic.co/integrations
| kubernetes.system.metrics.enabled | bool | `true` | enable system metric stream (kubelet) [ref](https://www.elastic.co/docs/current/integrations/kubernetes/kubelet#system) |
| kubernetes.system.metrics.vars | object | `{}` | system metric stream vars |

### 3 - User Extra Integrations
### 4 - System integration
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| system.enabled | bool | `false` | enable System integration. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the default false? I feel like most will want this on, it brings immediate value to the Observability section in Kibana.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More than happy to have it by default as true, it also makes sense to me. @nimarezainia what do you think on that? 🙂

| system.output | string | `"default"` | name of the output used in System integration. Note that this output needs to be defined in [outputs](#1-outputs) |
| system.namespace | string | `"default"` | output namespace |
| system.authLogs.enabled | bool | `true` | enable auth logs |
| system.authLogs.vars | object | `{}` | override default variables of auth logs stream. Look in the [values.schema.json](values.schema.json) to see the available variables |
| system.syslog.enabled | bool | `true` | enable syslog |
| system.syslog.vars | object | `{}` | override default variables of syslog stream. Look in the [values.schema.json](values.schema.json) to see the available variables |
| system.metrics.enabled | bool | `true` | enable metrics |

### 5 - User Extra Integrations
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| extraIntegrations | object | `{}` | extra [user-defined integrations](https://www.elastic.co/guide/en/fleet/current/elastic-agent-input-configuration.html) to be added to the Elastic Agent An example can be found [here](./examples/nginx-custom-integration/README.md) |

### 3 - Elastic-Agent Configuration
### 6 - Elastic-Agent Configuration
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| agent.version | string | `"9.0.0"` | elastic-agent version |
Expand All @@ -138,7 +150,7 @@ The chart built-in [kubernetes integration](https://docs.elastic.co/integrations
| agent.unprivileged | bool | `false` | enable unprivileged mode |
| agent.presets | map[string]{} | `{ "perNode" : {...}, "clusterWide": {...}, "ksmSharded": {...} }` | Map of deployment presets for the Elastic Agent. The key of the map is the name of the preset. See more for the presets required by the built-in Kubernetes integration [here](./values.yaml) |

### 3.1 - Elastic-Agent Managed Configuration
### 6.1 - Elastic-Agent Managed Configuration
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| agent.fleet.enabled | bool | `false` | enable elastic-agent managed |
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/elastic-agent/examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
Here is a collection of example configurations for the eck-integrations chart.

- [Kubernetes - defaults](kubernetes-default/README.md)
- [System - custom authlogs path](kubernetes-default/README.md)
- [Kubernetes - only container logs](kubernetes-only-logs/README.md)
- [Kubernetes - only hints autodiscover](kubernetes-hints-autodiscover/README.md)
- [Custom Integration - nginx](nginx-custom-integration/README.md)
Expand Down
25 changes: 25 additions & 0 deletions deploy/helm/elastic-agent/examples/system-custom-auth-paths/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Example: System Integration with custom auth log paths

In this example we install the built-in `system` integration and specify custom paths for the auth logs stream (see [agent-system-values.yaml](agent-system-values.yaml)).

## Prerequisites:
1. A k8s secret that contains the connection details to an Elasticsearch cluster such as the URL and the API key ([Kibana - Creating API Keys](https://www.elastic.co/guide/en/kibana/current/api-keys.html)):
```console
kubectl create secret generic es-api-secret \
--from-literal=api_key=... \
--from-literal=url=...
```

2. `system` integration assets installed through Kibana ([Kibana - Install and uninstall Elastic Agent integration assets](https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html))

## Run:
```console
helm install elastic-agent ../../ \
-f ./agent-system-values.yaml \
--set outputs.default.type=ESSecretAuthAPI \
--set outputs.default.secretName=es-api-secret
```

## Validate:

1. The Kibana `system`-related dashboards should start showing up the respective info.
13 changes: 13 additions & 0 deletions deploy/helm/elastic-agent/examples/system-custom-auth-paths/agent-system-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
system:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have rendered examples for this, right? I thought the mage target would error in this case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I thought I added it but this never happened, but the CI should fail yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I suspect that this is because the file was never versioned and the check-ci apparently fails on changes of versioned files?!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it fail anyway if it can't find the file? I didn't know the examples mage target was aware of versioning.

Anyway, this isn't a blocker for this PR, but we should make sure the check actually catches issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current CI checks flow is the following:

  • render helm examples
  • check for git diffs; the assumption here was that it failed also on unversioned files but evidently that's not the case

100% agreed this needs to be extended

enabled: true
authLogs:
vars:
paths:
- /var/log/custom_auth.log
syslog:
vars:
paths:
- /var/log/custom_syslog.log

agent:
unprivileged: true
Loading