Skip to content

Commit

Permalink
Merge pull request containerd#8722 from marquiz/devel/cgroup-driver-a…
Browse files Browse the repository at this point in the history
…utoconfig

cri: implement RuntimeConfig rpc
  • Loading branch information
fuweid authored Aug 4, 2023
2 parents 40c85d6 + c62abcb commit 2b2195c
Show file tree
Hide file tree
Showing 85 changed files with 11,283 additions and 494 deletions.
16 changes: 16 additions & 0 deletions docs/cri/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,22 @@ cgroupDriver: "systemd"
kubeadm users should also see [the kubeadm documentation](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
> Note: Kubernetes v1.28 supports automatic detection of the cgroup driver as
> an alpha feature. With the `KubeletCgroupDriverFromCRI` kubelet feature gate
> enabled, the kubelet automatically detects the cgroup driver from the CRI
> runtime and the `KubeletConfiguration` configuration step above is not
> needed.
>
> When determining the cgroup driver, containerd uses the `SystemdCgroup`
> setting from runc-based runtime classes, starting from the default runtime
> class. If no runc-based runtime classes have been configured containerd
> relies on auto-detection based on determining if systemd is running.
> Note that all runc-based runtime classes should be configured to have the
> same `SystemdCgroup` setting in order to avoid unexpected behavior.
>
> The automatic cgroup driver configuration for kubelet feature is supported in
> containerd v2.0 and later.

### Snapshotter

The default snapshotter is set to `overlayfs` (akin to Docker's `overlay2` storage driver):
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ require (
k8s.io/apiserver v0.26.2
k8s.io/client-go v0.26.2
k8s.io/component-base v0.26.2
k8s.io/cri-api v0.27.1
k8s.io/cri-api v0.28.0-beta.0
k8s.io/klog/v2 v2.90.1
k8s.io/utils v0.0.0-20230220204549-a5ecb0141aa5
)
Expand Down
7 changes: 4 additions & 3 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -874,7 +874,7 @@ github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40T
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
Expand All @@ -887,6 +887,7 @@ github.com/sclevine/spec v1.2.0/go.mod h1:W4J29eT/Kzv7/b9IWLB055Z+qvVC9vt0Arko24
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/seccomp/libseccomp-golang v0.9.1/go.mod h1:GbW5+tmTXfcxTToHLXlScSlAvWlF4P2Ca7zGrPiEpWo=
github.com/seccomp/libseccomp-golang v0.9.2-0.20210429002308-3879420cc921/go.mod h1:JA8cRccbGaA1s33RQf7Y1+q9gHmZX1yB/z9WDN1C6fg=
github.com/seccomp/libseccomp-golang v0.9.2-0.20220502022130-f33da4d89646 h1:RpforrEYXWkmGwJHIGnLZ3tTWStkjVVstwzNGqxX2Ds=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.0.4-0.20170822132746-89742aefa4b2/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
Expand Down Expand Up @@ -1634,8 +1635,8 @@ k8s.io/cri-api v0.20.1/go.mod h1:2JRbKt+BFLTjtrILYVqQK5jqhI+XNdF6UiGMgczeBCI=
k8s.io/cri-api v0.20.4/go.mod h1:2JRbKt+BFLTjtrILYVqQK5jqhI+XNdF6UiGMgczeBCI=
k8s.io/cri-api v0.20.6/go.mod h1:ew44AjNXwyn1s0U4xCKGodU7J1HzBeZ1MpGrpa5r8Yc=
k8s.io/cri-api v0.23.1/go.mod h1:REJE3PSU0h/LOV1APBrupxrEJqnoxZC8KWzkBUHwrK4=
k8s.io/cri-api v0.27.1 h1:KWO+U8MfI9drXB/P4oU9VchaWYOlwDglJZVHWMpTT3Q=
k8s.io/cri-api v0.27.1/go.mod h1:+Ts/AVYbIo04S86XbTD73UPp/DkTiYxtsFeOFEu32L0=
k8s.io/cri-api v0.28.0-beta.0 h1:JGtnKV4s7/1Pl2dWJX5s/Cl2074Fgry5TGLpDYkEapE=
k8s.io/cri-api v0.28.0-beta.0/go.mod h1:PgM+VelU7VKINUeaNLdE4fElKXfORIfTRNRM5wFBRCw=
k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
k8s.io/gengo v0.0.0-20200428234225-8167cfdcfc14/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
k8s.io/gengo v0.0.0-20201113003025-83324d819ded/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
Expand Down
5 changes: 3 additions & 2 deletions integration/client/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1539,8 +1539,9 @@ github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6So
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
Expand Down Expand Up @@ -2572,7 +2573,7 @@ k8s.io/code-generator v0.19.7/go.mod h1:lwEq3YnLYb/7uVXLorOJfxg+cUu2oihFhHZ0n9NI
k8s.io/component-base v0.26.2/go.mod h1:DxbuIe9M3IZPRxPIzhch2m1eT7uFrSBJUBuVCQEBivs=
k8s.io/cri-api v0.20.1/go.mod h1:2JRbKt+BFLTjtrILYVqQK5jqhI+XNdF6UiGMgczeBCI=
k8s.io/cri-api v0.25.3/go.mod h1:riC/P0yOGUf2K1735wW+CXs1aY2ctBgePtnnoFLd0dU=
k8s.io/cri-api v0.27.1/go.mod h1:+Ts/AVYbIo04S86XbTD73UPp/DkTiYxtsFeOFEu32L0=
k8s.io/cri-api v0.28.0-beta.0/go.mod h1:PgM+VelU7VKINUeaNLdE4fElKXfORIfTRNRM5wFBRCw=
k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
k8s.io/gengo v0.0.0-20200428234225-8167cfdcfc14/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
k8s.io/gengo v0.0.0-20201113003025-83324d819ded/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
Expand Down
16 changes: 16 additions & 0 deletions pkg/cri/instrument/instrumented_service.go
Original file line number Diff line number Diff line change
Expand Up @@ -621,3 +621,19 @@ func (in *instrumentedService) ListPodSandboxMetrics(ctx context.Context, r *run
res, err = in.c.ListPodSandboxMetrics(ctx, r)
return res, errdefs.ToGRPC(err)
}

func (in *instrumentedService) RuntimeConfig(ctx context.Context, r *runtime.RuntimeConfigRequest) (res *runtime.RuntimeConfigResponse, err error) {
if err := in.checkInitialized(); err != nil {
return nil, err
}
log.G(ctx).Tracef("RuntimeConfig")
defer func() {
if err != nil {
log.G(ctx).WithError(err).Error("RuntimeConfig failed")
} else {
log.G(ctx).Tracef("RuntimeConfig returns config %+v", res)
}
}()
res, err = in.c.RuntimeConfig(ctx, r)
return res, errdefs.ToGRPC(err)
}
31 changes: 31 additions & 0 deletions pkg/cri/sbserver/runtime_config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sbserver

import (
"context"

runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

// RuntimeConfig returns configuration information of the runtime.
func (c *criService) RuntimeConfig(ctx context.Context, r *runtime.RuntimeConfigRequest) (*runtime.RuntimeConfigResponse, error) {
resp := &runtime.RuntimeConfigResponse{
Linux: c.getLinuxRuntimeConfig(ctx),
}
return resp, nil
}
81 changes: 81 additions & 0 deletions pkg/cri/sbserver/runtime_config_linux.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sbserver

import (
"context"
"sort"

"github.com/containerd/containerd/log"
runcoptions "github.com/containerd/containerd/runtime/v2/runc/options"
"github.com/opencontainers/runc/libcontainer/cgroups/systemd"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

func (c *criService) getLinuxRuntimeConfig(ctx context.Context) *runtime.LinuxRuntimeConfiguration {
return &runtime.LinuxRuntimeConfiguration{CgroupDriver: c.getCgroupDriver(ctx)}
}

func (c *criService) getCgroupDriver(ctx context.Context) runtime.CgroupDriver {
// Go through the runtime handlers in a predictable order, starting from the
// default handler, others sorted in alphabetical order
handlerNames := make([]string, 0, len(c.config.ContainerdConfig.Runtimes))
for n := range c.config.ContainerdConfig.Runtimes {
handlerNames = append(handlerNames, n)
}
sort.Slice(handlerNames, func(i, j int) bool {
if handlerNames[i] == c.config.ContainerdConfig.DefaultRuntimeName {
return true
}
if handlerNames[j] == c.config.ContainerdConfig.DefaultRuntimeName {
return false
}
return handlerNames[i] < handlerNames[j]
})

for _, handler := range handlerNames {
opts, err := generateRuntimeOptions(c.config.ContainerdConfig.Runtimes[handler])
if err != nil {
log.G(ctx).Debugf("failed to parse runtime handler options for %q", handler)
continue
}
if d, ok := getCgroupDriverFromRuntimeHandlerOpts(opts); ok {
return d
}
log.G(ctx).Debugf("runtime handler %q does not provide cgroup driver information", handler)
}

// If no runtime handlers have a setting, detect if systemd is running
d := runtime.CgroupDriver_CGROUPFS
if systemd.IsRunningSystemd() {
d = runtime.CgroupDriver_SYSTEMD
}
log.G(ctx).Debugf("no runtime handler provided cgroup driver setting, using auto-detected %s", runtime.CgroupDriver_name[int32(d)])
return d
}

func getCgroupDriverFromRuntimeHandlerOpts(opts interface{}) (runtime.CgroupDriver, bool) {
switch v := opts.(type) {
case *runcoptions.Options:
systemdCgroup := v.SystemdCgroup
if systemdCgroup {
return runtime.CgroupDriver_SYSTEMD, true
}
return runtime.CgroupDriver_CGROUPFS, true
}
return runtime.CgroupDriver_SYSTEMD, false
}
105 changes: 105 additions & 0 deletions pkg/cri/sbserver/runtime_config_linux_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sbserver

import (
"context"
"testing"

criconfig "github.com/containerd/containerd/pkg/cri/config"
"github.com/containerd/containerd/plugin"
"github.com/opencontainers/runc/libcontainer/cgroups/systemd"
"github.com/stretchr/testify/assert"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

func newFakeRuntimeConfig(runcV2, systemdCgroup bool) criconfig.Runtime {
r := criconfig.Runtime{Type: "default", Options: map[string]interface{}{}}
if runcV2 {
r.Type = plugin.RuntimeRuncV2
if systemdCgroup {
r.Options["SystemdCgroup"] = true
}
}
return r
}

func TestRuntimeConfig(t *testing.T) {
autoDetected := runtime.CgroupDriver_CGROUPFS
if systemd.IsRunningSystemd() {
autoDetected = runtime.CgroupDriver_SYSTEMD
}

for _, test := range []struct {
desc string
defaultRuntime string
runtimes map[string]criconfig.Runtime
expectedCgroupDriver runtime.CgroupDriver
}{
{
desc: "no runtimes",
expectedCgroupDriver: autoDetected,
},
{
desc: "non-runc runtime",
defaultRuntime: "non-runc",
runtimes: map[string]criconfig.Runtime{"non-runc": newFakeRuntimeConfig(false, false)},
expectedCgroupDriver: autoDetected,
},
{
desc: "no default, pick first in alphabetical order",
runtimes: map[string]criconfig.Runtime{
"non-runc": newFakeRuntimeConfig(false, false),
"runc-2": newFakeRuntimeConfig(true, true),
"runc": newFakeRuntimeConfig(true, false),
"non-runc-2": newFakeRuntimeConfig(false, false),
},
expectedCgroupDriver: runtime.CgroupDriver_CGROUPFS,
},
{
desc: "pick default, cgroupfs",
defaultRuntime: "runc-2",
runtimes: map[string]criconfig.Runtime{
"non-runc": newFakeRuntimeConfig(false, false),
"runc": newFakeRuntimeConfig(true, true),
"runc-2": newFakeRuntimeConfig(true, false),
},
expectedCgroupDriver: runtime.CgroupDriver_CGROUPFS,
},
{
desc: "pick default, systemd",
defaultRuntime: "runc-2",
runtimes: map[string]criconfig.Runtime{
"non-runc": newFakeRuntimeConfig(false, false),
"runc": newFakeRuntimeConfig(true, false),
"runc-2": newFakeRuntimeConfig(true, true),
},
expectedCgroupDriver: runtime.CgroupDriver_SYSTEMD,
},
} {
test := test
t.Run(test.desc, func(t *testing.T) {
c := newTestCRIService()
c.config.PluginConfig.ContainerdConfig.DefaultRuntimeName = test.defaultRuntime
c.config.PluginConfig.ContainerdConfig.Runtimes = test.runtimes

resp, err := c.RuntimeConfig(context.TODO(), &runtime.RuntimeConfigRequest{})
assert.NoError(t, err)
assert.Equal(t, test.expectedCgroupDriver, resp.Linux.CgroupDriver, "got unexpected cgroup driver")
})
}
}
29 changes: 29 additions & 0 deletions pkg/cri/sbserver/runtime_config_other.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
//go:build !linux

/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package sbserver

import (
"context"

runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

func (c *criService) getLinuxRuntimeConfig(ctx context.Context) *runtime.LinuxRuntimeConfiguration {
return nil
}
31 changes: 31 additions & 0 deletions pkg/cri/server/runtime_config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package server

import (
"context"

runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

// RuntimeConfig returns configuration information of the runtime.
func (c *criService) RuntimeConfig(ctx context.Context, r *runtime.RuntimeConfigRequest) (*runtime.RuntimeConfigResponse, error) {
resp := &runtime.RuntimeConfigResponse{
Linux: c.getLinuxRuntimeConfig(ctx),
}
return resp, nil
}
Loading

0 comments on commit 2b2195c

Please sign in to comment.