[helm]: integration config for scheduler and controller-manager conditions and containers logs

[pkoutsovasilis]

What does this PR do?

I was testing k8s standalone elastic-agent using the plain k8s manifest under deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml but I also tried to get the same results with the elastic-agent helm chart and I tried to replicate the produced data with the elastic-agent helm chart. This effort uncovered some yaml format issues and some inconsistencies for container logs which this PR handles

Why is it important?

Because it fixes bugs 🙂

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

N/A

How to test this PR locally

 mage integration:kubernetesMatrix

Related issues

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

Was these sections removed because it caused the events to not match the integration?

[swiatekm]

If we do want the container logs to be identical, shouldn't we simply copy the standalone config from here:

elastic-agent/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

Line 354 in b5b842d

- id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}

?

[pkoutsovasilis]

@swiatekm not entirely sure why the standalone agent manifest doesn't include them but this is what Kibana generates when you try to install the kubernetes integration which includes default values (snippet regards container logs)

      - id: filestream-container-logs-1b427a99-5d73-4e5c-a554-6183c5cf40f7
        revision: 1
        name: kubernetes-1
        type: filestream
        data_stream:
          namespace: default
        use_output: default
        package_policy_id: 1b427a99-5d73-4e5c-a554-6183c5cf40f7
        streams:
          - id: >-
              kubernetes-container-logs-${kubernetes.pod.name}-${kubernetes.container.id}
            data_stream:
              dataset: kubernetes.container_logs
            paths:
              - '/var/log/containers/*${kubernetes.container.id}.log'
            prospector:
              scanner:
                fingerprint.enabled: true
                symlinks: true
            file_identity.fingerprint: null
            parsers:
              - container:
                  stream: all
                  format: auto
            processors:
              - add_fields:
                  target: kubernetes
                  fields:
                    annotations.elastic_co/dataset: '${kubernetes.annotations.elastic.co/dataset|""}'
                    annotations.elastic_co/namespace: '${kubernetes.annotations.elastic.co/namespace|""}'
                    annotations.elastic_co/preserve_original_event: >-
                      ${kubernetes.annotations.elastic.co/preserve_original_event|""}
              - drop_fields:
                  fields:
                    - kubernetes.annotations.elastic_co/dataset
                  when:
                    equals:
                      kubernetes.annotations.elastic_co/dataset: ''
                  ignore_missing: true
              - drop_fields:
                  fields:
                    - kubernetes.annotations.elastic_co/namespace
                  when:
                    equals:
                      kubernetes.annotations.elastic_co/namespace: ''
                  ignore_missing: true
              - drop_fields:
                  fields:
                    - kubernetes.annotations.elastic_co/preserve_original_event
                  when:
                    equals:
                      kubernetes.annotations.elastic_co/preserve_original_event: ''
                  ignore_missing: true
              - add_tags:
                  tags:
                    - preserve_original_event
                  when:
                    and:
                      - has_fields:
                          - >-
                            kubernetes.annotations.elastic_co/preserve_original_event
                      - regexp:
                          kubernetes.annotations.elastic_co/preserve_original_event: ^(?i)true$

So I think that it should not be a copy paste from there?!

@blakerouse thank you for catching that, I was experimenting with some config until I realised I need hostNetwork: true to make the node name appear instead of the pod name, and they shouldn't have been removed 🙂

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube