Revert "Support flattened data_stream.* fields"

[belimawr]

Reverts #3465 because it seems to be breaking some integration tests: #3741

[mergify]

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[cmacknz]

buildkite test it

[cmacknz]

The previous version of the test failed with:

Failed to uninstall agent  [9s] Error: error uninstalling agent: failed to remove installation directory (C:\\Program Files\\Elastic\\Agent): timed out while removing \"C:\\\\Program Files\\\\Elastic\\\\Agent\". Last error: remove C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-3238b7\\elastic-agent.exe: Access is denied.\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.12/fleet-troubleshooting.html\n

This is an unrelated problem.

[cmacknz]

Reverting this reveals a different failure:

        Elastic Agent has been successfully installed.
    monitoring_logs_test.go:70: created policy: 905338ce-b6d3-4920-ab57-384c59eddb36
    fixture.go:505: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    monitoring_logs_test.go:77: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/monitoring_logs_test.go:77
        	Error:      	Received unexpected error:
        	            	error unmarshaling response: error unmarshalling JSON for string 'null': strconv.ParseInt: parsing "null": invalid syntax
        	Test:       	TestMonitoringLogsShipped
    fixture_install.go:127: [test TestMonitoringLogsShipped] Inside fixture cleanup function
    fixture_install.go:142: collecting diagnostics; test failed
    fixture.go:505: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /home/ubuntu/agent/build/diagnostics/TestMonitoringLogsShipped-diagnostics-2023-11-10T20:13:50Z.zip]
    fixture.go:505: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
--- FAIL: TestMonitoringLogsShipped (56.95s)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-13T12:57:09.639+0000

• Duration: 62 min 23 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pchila]

/test

[elastic-sonarqube]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b revert-3465-flattened-datastreams upstream/revert-3465-flattened-datastreams
git merge upstream/main
git push upstream revert-3465-flattened-datastreams

[belimawr]

The fix for the root cause that triggered this revert is here: #3765.