operator-sdk 1.28.0 - Problem generating CRD with corev1.ResourceRequirements type

[gaohoward]

Bug Report

What did you do?

We are upgrading our operator-sdk to 1.28.0 (from 1.25.0). In our CRD types we have a field that is of corev1.ResourceRequirements type.

What did you expect to see?

The generated CRD should be deployed without error.

What did you see instead? Under which circumstances?

When we deploy the generated CRDs to kubernetes it cannot pass
the schema validation. The error message is look like
Error from server (Invalid): error when creating "deploy.yaml": CustomResourceDefinition.apiextensions.k8s.io "activemqartemises.broker.amq.io" is invalid: [spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[deploymentPlan].properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set

Environment

Operator type:

/language go

Kubernetes cluster type:

minikube v1.29.0 on Fedora 37

$ operator-sdk version

perator-sdk version: "v1.28.0", commit: "484013d1865c35df2bc5dfea0ab6ea6b434adefa", kubernetes version: "1.26.0", go version: "go1.19.6", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.19.3 linux/amd64

$ kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

No solution so far but it looks like the resourceRequirements.Claims field (https://github.com/kubernetes/api/blob/master/core/v1/types.go#L2356) causes the problem. This field is generated the following piece in the CRD

                      claims:
                        description: "Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container. \n This is an alpha field and requires enabling the DynamicResourceAllocation feature gate. \n This field is immutable."
                        items:
                          description: ResourceClaim references one entry in PodSpec.ResourceClaims.
                          properties:
                            name:
                              description: Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container.
                              type: string
                          required:
                          - name
                          type: object
                        type: array
                        x-kubernetes-list-type: set

and if I remove x-kubernetes-list-type: set manually and the CRD can be deployed without error.

Additional context

[Murphychih]

I got same problem at operator-sdk 1.28.0

[gaohoward]

Hi,

This issue blocks our migration. Any advice how to solve it?
Thanks

[bimtauer]

Same issue here. Any validation markers that can fix this?

[everettraven]

From what I could tell, it seems like this would be an issue with the version of controller-gen that is being used. It looks like the corev1.ResourceRequirements type is using some markers processed incorrectly by controller-gen v0.11.1.

There was an issue created in the controller-tools repo for this same thing: kubernetes-sigs/controller-tools#772

It looks like the workaround for now is to manually bump the controller-gen version in your Makefile to v0.11.2

[varshaprasad96]

We may have to bump controller-gen in the scaffolded projects to solve this.

[gaohoward]

Thanks @everettraven !

[everettraven]

We may have to bump controller-gen in the scaffolded projects to solve this.

@varshaprasad96 +1 - I think we will need to do this in Kubebuilder though and then pull that in

[varshaprasad96]

An upstream issue is created to address this: kubernetes-sigs/kubebuilder#3313. Adding this to backlog and keeping it open to track till the upstream KB is fixed.

[legolego621]

Hello, is there any movement on this issue?
I get this error in operator-sdk version: "v1.30.0"

I have changed controller-gen version, but I have this problem.

$ bin/controller-gen --version
Version: v0.11.2
$ make docker-build docker-push IMG="$IMAGE_OPERATOR"
$ make deploy IMG="$IMAGE_OPERATOR"
The CustomResourceDefinition "exampleapp.apps.test.local" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
make: *** [Makefile:167: deploy] Error 1

[legolego621]

Hello, is there any movement on this issue? I get this error in operator-sdk version: "v1.30.0"

I have changed controller-gen version, but I have this problem.

$ bin/controller-gen --version
Version: v0.11.2
$ make docker-build docker-push IMG="$IMAGE_OPERATOR"
$ make deploy IMG="$IMAGE_OPERATOR"
The CustomResourceDefinition "exampleapp.apps.test.local" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
make: *** [Makefile:167: deploy] Error 1

Fixed for me
kubernetes-sigs/controller-tools#772 (comment)

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.