Kubernetes metadata discovery broken on older containerd versions

[brancz]

I noticed on the demo instance that even processes that should have Kubernetes metadata do not have it. For example Parca itself:

Enabling debug logging revealed what I believe to be the culprit:

level=debug name=parca-agent ts=2022-12-20T17:37:42.319381051Z caller=kubernetes.go:195 discovery=scw-parca-demo-default-88ff9f23d1d24f079f61fa3 msg="skipping pod, cannot find pid" namespace=parca pod=parca-agent-blb2b err="failed to get container status, request: &ContainerStatusRequest{ContainerId:fc3c779694064e6f0409913c67370204e54efafa3d6629a928cefb51a709a0de,Verbose:true,}: rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

It looks like we recently switched to using the v1 API only (#1118), but there are still lots of containerd v1.5.x versions out there being used that only serve the v1alpha2 API. Since containerd v1.6.x supports both versions of the API, I would suggest that we revert to v1.25 of cri-api (which still has both definitions), and for now always use the v1alpha2 API.

I've also tested this on a cluster that uses containerd 1.6.x and it works as expected there.

[maxbrunet]

containerd v1.5 should be supported until January 28, 2023, but v1.5.8 has support for v1, any other release is more than a year old.

In my opinion, if the issue appeared on a Kubernetes clusterr older than v1.22, it should be upgraded, and if it is v1.23 or above, its containerd version should be upgraded to v1.5.8+ to meet the requirements.

https://github.com/containerd/containerd/blob/main/RELEASES.md#support-horizon
https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support

[brancz]

I think I’d agree, but our own demo cluster has this issue so it seems that there are plenty of these versions around. The cluster was even pretty recently created on a managed cluster offering. I think we should do our utmost best to support maximum number of versions in use.

[maxbrunet]

Issues that probably need fixing, at least from a security stand point. I would suggest checking the node images are up-to-date or reach out to the service provider (I guess we are talking about Scaleway) to know why the minimum requirements are not respected and if they can fix it, you are right, it is their responsibility.

[brancz]

I did some digging and Kubernetes 1.26, which was released two weeks ago, dropped support for v1alpha2, and therefore cri-o and containerd are removing support as well. Having a skew of 4 minor Kubernetes versions supported is kind of the norm within the Kubernetes community, so I think you're right, we're essentially also dropping support for Kubernetes 1.22 and lower. 1.23 and newer clusters should have a newer containerd version anyway.

Closing as won't fix.

[kakkoyun]

Pinning to issue for more visibility. In case someone else comes across it.