Support flattened data_stream.* fields #3465

belimawr · 2023-09-22T13:26:19Z

What does this PR do?

An input configuration supports flattened fields, however the 'data_stream' field was not being correctly decoded when flattened. This commit fixes this issue.

Why is it important?

It increases consistency within the Elastic-Agent configuration and our other projects, like Beats, because they all support flattened configuration keys.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

~~## Author's Checklist~~

How to test this PR locally

Deploy a standalone Elastic-Agent with a configuration like the following.

outputs:
  default:
    type: elasticsearch
    hosts:
      - "${ELASTICSEARCH_HOST}"
    username: "${ELASTICSEARCH_USERNAME}"
    password: "${ELASTICSEARCH_PASSWORD}"

inputs:
  - type: filestream
    id: elastic-agent-input-id
    streams:
      - id: filestream-input-id-1
        data_stream:
          dataset: generic
        data_stream.namespace: "flattened-namespace"
        data_stream.type: "logs"
        paths:
          - /tmp/log.log

Make sure some fields from data_stream.* are flattened. It is even better if there is a mix
Look at the ingested data and make sure the fields correctly set.

Related issues

~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2023-09-22T13:27:18Z

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-09-22T13:35:34Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-08T07:30:24.032+0000
Duration: 13 min 7 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-09-22T14:00:34Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.824% (`84/85`)	👍
Files	66.885% (`204/305`)	👍
Classes	65.95% (`368/558`)	👍
Methods	53.105% (`1163/2190`)	👍 0.064
Lines	39.43% (`13708/34765`)	👍 0.059
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-10-09T11:09:02Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

belimawr · 2023-10-09T13:50:13Z

testing/integration/datastreams_test.go

+	info := define.Require(t, define.Requirements{
+		Local: false,
+		Stack: &define.Stack{
+			Version: version.Agent + "-SNAPSHOT",


CI was failing because the stack was too old, so I "hardcoded" the version at least to make sure the tests are passing

belimawr · 2023-10-09T14:51:26Z

/test

faec

A few nits for code clarity but looks good to me!

pkg/component/config.go

belimawr · 2023-10-10T13:18:23Z

/test

testing/integration/datastreams_test.go

belimawr · 2023-10-27T14:00:00Z

/test

mergify · 2023-11-01T18:40:22Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b flattened-datastreams upstream/flattened-datastreams
git merge upstream/main
git push upstream flattened-datastreams

belimawr · 2023-11-06T18:45:42Z

fleet-ci/pr-merge is stuck 🤦‍♂️

An input configuration supports flattened fields, however the 'data_stream' field was not being correctly decoded when flattened. This commit fixes this issue. Some small additions and refactoring are also implemented in the integration test framework as well as some more detailed documentation.

Co-authored-by: Paolo Chilà <[email protected]>

Fix all failing tests

belimawr · 2023-11-06T18:46:59Z

I rebased onto main to re-trigger the tests

cmacknz

You should update the reference YAML to include an example of this style of configuration, I would recommend modifying the sample log input:

elastic-agent/elastic-agent.yml

Lines 47 to 58 in dba2166

    
           #   # Collecting log files 
        
           #   - type: filestream 
        
           #     # Input ID allowing Elastic Agent to track the state of this input. Must be unique. 
        
           #     id: your-input-id 
        
           #     streams: 
        
           #       # Stream ID for this data stream allowing Filebeat to track the state of the ingested files. Must be unique. 
        
           #       # Each filestream data stream creates a separate instance of the Filebeat filestream input. 
        
           #       - id: your-filestream-stream-id 
        
           #         data_stream: 
        
           #           dataset: generic 
        
           #         paths: 
        
           #           - /var/log/*.log

If the documentation isn't updated nobody is going to use this configuration style :)

belimawr · 2023-11-07T07:57:03Z

You should update the reference YAML to include an example of this style of configuration, I would recommend modifying the sample log input:

If the documentation isn't updated nobody is going to use this configuration style :)

We already have examples since about 3 years ago 😱

elastic-agent/elastic-agent.reference.yml

Lines 24 to 39 in e9be070

    
           data_stream.namespace: default 
        
           use_output: default 
        
           streams: 
        
             - metricsets: 
        
               - cpu 
        
               # Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes 
        
               data_stream.dataset: system.cpu 
        
             - metricsets: 
        
               - memory 
        
               data_stream.dataset: system.memory 
        
             - metricsets: 
        
               - network 
        
               data_stream.dataset: system.network 
        
             - metricsets: 
        
               - filesystem 
        
               data_stream.dataset: system.filesystem

Anyways, I'll edit the filestream example as well.

Revert my last chances in the `elastic-agent.reference.yml` because `mage check` does not like it.

belimawr · 2023-11-08T07:33:46Z

@cmacknz mage check did not like my last changes in the configuration :/

Aside from the flattened examples I pointed out before we also have it shown in our documentation: https://github.com/elastic/ingest-docs/blob/bab384ba275f4ae34696cedbdc1221383b64e1c9/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-configuration.asciidoc?plain=1#L37-L50

I believe we have a good mix of syntax example.

I confess I'm puzzled about mage check only editing the commented out section of the reference configuration 🤔

elastic-sonarqube · 2023-11-08T07:54:58Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

78.6% Coverage
0.0% Duplication

This reverts commit b0b8e85.

belimawr added the Team:Elastic-Agent Label for the Agent team label Sep 22, 2023

mergify bot assigned belimawr Sep 22, 2023

mergify bot added the backport-skip label Sep 22, 2023

belimawr force-pushed the flattened-datastreams branch from d0c797a to 9da835e Compare September 22, 2023 13:33

belimawr force-pushed the flattened-datastreams branch 2 times, most recently from 3f81294 to a91af36 Compare September 22, 2023 16:30

belimawr mentioned this pull request Sep 25, 2023

Add logging configuration example to the Configure inputs page elastic/ingest-docs#489

Merged

belimawr force-pushed the flattened-datastreams branch 5 times, most recently from 07020fb to 2b2bcad Compare October 6, 2023 14:29

belimawr force-pushed the flattened-datastreams branch 2 times, most recently from 9812846 to da565c3 Compare October 9, 2023 11:07

belimawr marked this pull request as ready for review October 9, 2023 11:09

belimawr requested a review from a team as a code owner October 9, 2023 11:09

belimawr requested review from AndersonQ and faec October 9, 2023 11:09

belimawr commented Oct 9, 2023

View reviewed changes

faec approved these changes Oct 9, 2023

View reviewed changes

pkg/component/config.go Outdated Show resolved Hide resolved

pkg/component/config.go Outdated Show resolved Hide resolved

pkg/component/config.go Show resolved Hide resolved

pkg/component/config.go Outdated Show resolved Hide resolved

belimawr requested a review from faec October 10, 2023 08:56

cmacknz reviewed Oct 10, 2023

View reviewed changes

testing/integration/datastreams_test.go Outdated Show resolved Hide resolved

belimawr force-pushed the flattened-datastreams branch from 51bd99a to 92b25f5 Compare October 11, 2023 14:57

belimawr requested a review from pchila October 11, 2023 16:38

belimawr force-pushed the flattened-datastreams branch 2 times, most recently from 99f9579 to c743823 Compare October 27, 2023 11:19

belimawr mentioned this pull request Oct 27, 2023

Fix data race when accessing unversionedHome #3669

Closed

2 tasks

belimawr force-pushed the flattened-datastreams branch from c743823 to 71046b2 Compare November 3, 2023 15:08

belimawr and others added 9 commits November 6, 2023 19:46

make CI Happy

79e9f0d

Better documentation, small improvements, mage check

c823617

PR improvements

f9aece8

refactor tests to reuse stack and Elastic-Agent

8f1ab98

remove standalone test

9c6e325

Apply suggestions from code review

3706f6c

Co-authored-by: Paolo Chilà <[email protected]>

PR improvements

92b258e

Fix failing tests

a1c202c

Fix all failing tests

belimawr force-pushed the flattened-datastreams branch from 71046b2 to a1c202c Compare November 6, 2023 18:46

cmacknz reviewed Nov 6, 2023

View reviewed changes

Update docs

c4e64c4

cmacknz approved these changes Nov 7, 2023

View reviewed changes

revert edit

a3f54f9

Revert my last chances in the `elastic-agent.reference.yml` because `mage check` does not like it.

belimawr merged commit b0b8e85 into elastic:main Nov 8, 2023
12 checks passed

blakerouse mentioned this pull request Nov 8, 2023

Switch to CFT region and add more robust tracking and cleanup of stacks. #3701

Merged

4 tasks

cmacknz mentioned this pull request Nov 10, 2023

TestLogIngestionFleetManaged/Monitoring_logs_are_shipped is flaky #3741

Open

belimawr added a commit that referenced this pull request Nov 10, 2023

Revert "Support flattened data_stream.* fields (#3465)"

3238b7c

This reverts commit b0b8e85.

belimawr mentioned this pull request Nov 10, 2023

Revert "Support flattened data_stream.* fields" #3743

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support flattened data_stream.* fields #3465

Support flattened data_stream.* fields #3465

belimawr commented Sep 22, 2023 •

edited

Loading

mergify bot commented Sep 22, 2023

elasticmachine commented Sep 22, 2023 •

edited

Loading

Build stats

elasticmachine commented Sep 22, 2023 •

edited

Loading

elasticmachine commented Oct 9, 2023

belimawr Oct 9, 2023

belimawr commented Oct 9, 2023

faec left a comment

belimawr commented Oct 10, 2023

belimawr commented Oct 27, 2023

mergify bot commented Nov 1, 2023

belimawr commented Nov 6, 2023

belimawr commented Nov 6, 2023

cmacknz left a comment

belimawr commented Nov 7, 2023

belimawr commented Nov 8, 2023

elastic-sonarqube bot commented Nov 8, 2023

	# # Collecting log files
	# - type: filestream
	# # Input ID allowing Elastic Agent to track the state of this input. Must be unique.
	# id: your-input-id
	# streams:
	# # Stream ID for this data stream allowing Filebeat to track the state of the ingested files. Must be unique.
	# # Each filestream data stream creates a separate instance of the Filebeat filestream input.
	# - id: your-filestream-stream-id
	# data_stream:
	# dataset: generic
	# paths:
	# - /var/log/*.log

Support flattened data_stream.* fields #3465

Support flattened data_stream.* fields #3465

Conversation

belimawr commented Sep 22, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Questions to ask yourself

mergify bot commented Sep 22, 2023

elasticmachine commented Sep 22, 2023 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Sep 22, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Oct 9, 2023

belimawr Oct 9, 2023

Choose a reason for hiding this comment

belimawr commented Oct 9, 2023

faec left a comment

Choose a reason for hiding this comment

belimawr commented Oct 10, 2023

belimawr commented Oct 27, 2023

mergify bot commented Nov 1, 2023

belimawr commented Nov 6, 2023

belimawr commented Nov 6, 2023

cmacknz left a comment

Choose a reason for hiding this comment

belimawr commented Nov 7, 2023

belimawr commented Nov 8, 2023

elastic-sonarqube bot commented Nov 8, 2023

belimawr commented Sep 22, 2023 •

edited

Loading

elasticmachine commented Sep 22, 2023 •

edited

Loading

elasticmachine commented Sep 22, 2023 •

edited

Loading