feat(cmd/agent): add fallback X.509 trusted roots

[hainenber]

PR Description

Add fallback X.509 trusted roots to grafana-agent and grafana-agent-flow binaries.

Risk is mild to empty as the functionality is only activated via injecting env var GODEBUG=x509usefallbackroots=1 to the binary during runtime.

Which issue(s) this PR fixes

Implements #6003

Notes to the Reviewer

PR Checklist

• CHANGELOG.md updated

[ptodev]

Thanks @hainenber, LGTM. I suppose the main worry with this is that those certificates may no longer be trusted one day. If the users update their Agent binary regularly, this shouldn't be a problem.

Also, apparently the GODEBUG env var has to be set for any of this to take effect, so I think we can assume that we won't impact users negatively if they don't use the feature.

cc @clayton-cornell it might be good to document interesting uses of GODEBUG and other Go env vars (such as the ones for proxying). Maybe we can add a new section under Command-line interface? I could raise an issue for this if you'd like me to.

[clayton-cornell]

This isn't documented anywhere (that I can find). Is this something we should add to the HTTP or TLS docs?

[ptodev]

In the block configuration we normally only document the arguments which are in the block. I haven't seen documentation about environment variables in block documentation. I think we should make a special page for the env vars and link to it from the block docs.

@jkroepke could you please let us know which Agent components you are using in this case? And why is it not sufficient to use an argument such as ca_pem in the tls config block?

[jkroepke]

docs for

GODEBUG=x509usefallbackroots=1

https://pkg.go.dev/crypto/x509#SetFallbackRoots

I'm plan to use it for

• module.http
• remote.http
• prometheus.remote_write
• loki.write

And why is it not sufficient to use an argument such as ca_pem in the tls config block?

The ca_pem block is sufficient. But then I have to ship the whole ca-certificates pem file together with the setup. Using the fallback X.509 trusted roots from go also fits well and would eliminate the requirement of shipping a own ca pem file.

An other benefit is if the root CAs are embedded:

• grafana-agent could run in a FROM scratch docker image which reduce the attack surface to a minimum. Since grafana may has very high permissions in a Kubernetes cluster, reducing the dependencies to a minimum would increase the security as well.

Configure the ca_pem, but using the trust ca-certificates sounds silly for first time, but using a custom ca certificates file skips the system verifier. The issue with the system verifier under Windows is that go has no capabilities to configure flags for the Windows Crypto API.

Context: golang/go#63238 (comment)

[ptodev]

@jkroepke Thanks for clarifying - this also confirms what I inferred from the issue so far.

@clayton-cornell If we are to document such env vars, we would also need to document how they could interfere with arguments in the River config, and which one takes precedence. I could raise a separate issue for this? I don't think it's a must for now. In the past, I've seen this issue mostly with the HTTP_PROXY env var.

[clayton-cornell]

@ptodev I'd say drop it into an issue and backlog it for a future doc improvement. It's not critical right now... it's just something I wondered about as I read through the current change, and it would certainly be helpful to have this added at some point.

[ptodev]

@clayton-cornell I created a new issue - #6372