Integration tests on BuikdKite are getting stuck for hours

[belimawr]

Sometimes our integration tests on Buildkite are getting stuck for hours, the logs do not have a clear sign of anything going wrong, they seem to just stop executing at some point.

This is an example of a build that is stuck (as the time of writing):https://buildkite.com/elastic/elastic-agent/builds/5196#018c1836-e9a1-4d21-aac7-8f131760829f
PR: #3829

The logs just stop:

2023-11-29 00:31:49 UTC	>>> (linux-arm64-ubuntu-2204) Test output (sudo) (stdout): >> go test: remote-linux-arm64-ubuntu-2204-sudo.integration Test Passed
2023-11-29 00:35:36 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): ok  	github.com/elastic/elastic-agent/testing/integration	3573.590s
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === Skipped
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestProxyURL_RemoveProxyFromThePolicy (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): proxy_url_test.go:308: Flaky test: https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestProxyURL_EnrollProxyAndEmptyProxyInThePolicy (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): proxy_url_test.go:141: Flaky test: https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestFQDN (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): fqdn_test.go:42: Flaky test, see https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestProxyURL_NoEnrollProxyAndProxyInThePolicy (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): proxy_url_test.go:245: Flaky test: https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestProxyURL_ProxyInThePolicyTakesPrecedence (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): proxy_url_test.go:186: Flaky test: https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): === SKIP: testing/integration TestProxyURL_EnrollProxyAndNoProxyInThePolicy (0.00s)
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): proxy_url_test.go:97: Flaky test: https://github.com/elastic/elastic-agent/issues/3154
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): DONE 36 tests, 6 skipped in 3628.248s
2023-11-29 00:35:37 UTC	>>> (linux-amd64-ubuntu-2204) Test output (sudo) (stdout): >> go test: remote-linux-amd64-ubuntu-2204-sudo.integration Test Passed

This specific one has been running for 8h+

[pierrehilbert]

We have another issue for that, closing as a duplicate.