upgrades between CRDs are broken - kubectl explain not updated #95702

RamLavi · 2020-10-19T12:59:13Z

What happened:
I'm getting an issue with kubectl explain (in my case networkaddonsconfig ) when upgrading from version with apiextensions.k8s.io/v1beta1 to apiextensions.k8s.io/v1, where DESCRIPTION is empty even if the updated crd has got the description fields updated.

$ kubectl explain examplecrds
KIND:     exampleCrd
VERSION:  crd.io/v1beta1

DESCRIPTION:
     <empty>

The solution I saw so far is to set PreserveUnknownFields:false , as also mentioned in this open issue on etcd-cluster-operator),
however I find the solution troubling, as PreserveUnknownFields is deprecated and the alternative field does not support the value False.

Is there an expected version where PreserveUnknownFields will be removed entirely?
Is there a solution/workaround what to do when that happens?

What you expected to happen:
kubectl explain <crd> should be updated with the new fields when applying the new CRD, and not present <empty> field.

$ kubectl explain examplecrds
KIND:     exampleCrd
VERSION:  crd.io/v1beta1

DESCRIPTION:
     examplecrd is the Schema for the examplecrds API

FIELDS:
   apiVersion	<string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind	<string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata	<Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   status	<map[string]>
     exampleCrdStatus is the status of the exampleCrd of a specific node

How to reproduce it (as minimally and precisely as possible):

apply crd with apiextensions.k8s.io/v1beta1 (added example yaml: crd_apiex_v1beta1.yaml)

kubectl apply -f crd_apiex_v1beta1.yaml

upgrade it by applying the new crd with new description fields and apiextensions.k8s.io/v1 (added example yaml: crd_apiex_v1.yaml)

kubectl apply -f crd_apiex_v1.yaml

To check workaround using deprecated field:

remove any existing crd

kubectl delete -f crd_apiex_v1.yaml

apply crd with apiextensions.k8s.io/v1beta1 (added example yaml: crd_apiex_v1beta1.yaml)

kubectl apply -f crd_apiex_v1beta1.yaml

upgrade it by applying the new crd with new description fields, apiextensions.k8s.io/v1 and deprecated field preserveUnknownFields (added example yaml: crd_apiex_v1_dep_workaround.yaml)

kubectl apply -f crd_apiex_v1_dep_workaround.yaml

Example yamls for easy recreated of problem:
crd_apiex_v1beta1.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: examplecrds.crd.io
spec:
  group: crd.io
  names:
    kind: exampleCrd
    listKind: exampleCrdList
    plural: examplecrds
    singular: examplecrd
  scope: Cluster
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        status:
          type: object
      type: object
  version: v1beta1
  versions:
    - name: v1beta1
      served: true
      storage: true

crd_apiex_v1.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: examplecrds.crd.io
spec:
  group: crd.io
  names:
    kind: exampleCrd
    listKind: exampleCrdList
    plural: examplecrds
    singular: examplecrd
  scope: Cluster
  versions:
    - name: v1beta1
      schema:
        openAPIV3Schema:
          description: 'examplecrd is the Schema for the examplecrds API'
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            status:
              description: 'exampleCrdStatus is the status of the exampleCrd of a specific node'
              type: object
          type: object
      served: true
      storage: true

crd_apiex_v1_dep_workaround.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: examplecrds.crd.io
spec:
  group: crd.io
  names:
    kind: exampleCrd
    listKind: exampleCrdList
    plural: examplecrds
    singular: examplecrd
  scope: Cluster
  versions:
    - name: v1beta1
      schema:
        openAPIV3Schema:
          description: 'examplecrd is the Schema for the examplecrds API'
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            status:
              description: 'exampleCrdStatus is the status of the exampleCrd of a specific node'
              type: object
          type: object
      served: true
      storage: true
  preserveUnknownFields: false

Anything else we need to know?:
I initially opened this in api-extentions but I think it's not regularly updated so also opened here. sorry for make it duplicate.

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):

NAME=Fedora
VERSION="31 (Workstation Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Kernel (e.g. uname -a):

Linux localhost.localdomain 5.5.9-200.fc31.x86_64 #1 SMP Thu Mar 12 13:55:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
Network plugin and version (if this is a network-related bug):
Others:

The text was updated successfully, but these errors were encountered:

RamLavi · 2020-10-19T13:02:29Z

/sig api-machinery

RamLavi · 2020-10-21T08:57:45Z

other projects using this workaround, and may break when top-level PreserveUnknownFields is removed entirely: SSP PR, CDI PR, and CNAO PR

liggitt · 2020-10-21T12:55:43Z

apiextensions.k8s.io/v1beta1 defaults spec.preserveUnknownFields to true for backwards compatibility with early CRD behavior.

No openapi v2 schema is published for CRDs that set that to true (or omit that field in v1beta1 and are defaulted to true). Publishing the openapi v2 schema would trigger kubectl to do client-side validation, which would fail to send API objects containing arbitrary fields which the API server would allow (and which the CRD indicated it wanted to preserve). kubectl explain does not work for CRDs that don't have an openapi v2 schema published.

The solution I saw so far is to set PreserveUnknownFields:false , as also mentioned in this open issue on etcd-cluster-operator),

If you have a v1beta1 CRD that set spec.preserveUnknownFields to true (or omitted it and was defaulted to true), modifying it by explicitly setting spec.preserveUnknownFields to false is the correct way to indicate you do not want to allow arbitrary fields, and to enable publishing the openapi schema.

however I find the solution troubling, as PreserveUnknownFields is deprecated and the alternative field does not support the value False.

In apiextensions.k8s.io/v1, spec.preserveUnknownFields can only be set to false in new CRDs, and defaults to false if unspecified, so if you create a new CRD via the v1 API you'll get the behavior you want. If you have an existing CRD, you can explicitly set it to false.

Is there an expected version where PreserveUnknownFields will be removed entirely?

No.

/close

k8s-ci-robot · 2020-10-21T12:55:57Z

@liggitt: Closing this issue.

In response to this:

apiextensions.k8s.io/v1beta1 defaults spec.preserveUnknownFields to true for backwards compatibility with early CRD behavior.

No openapi v2 schema is published for CRDs that set that to true (or omit that field in v1beta1 and are defaulted to true). Publishing the openapi v2 schema would trigger kubectl to do client-side validation, which would fail to send API objects containing arbitrary fields which the API server would allow (and which the CRD indicated it wanted to preserve). kubectl explain does not work for CRDs that don't have an openapi v2 schema published.

The solution I saw so far is to set PreserveUnknownFields:false , as also mentioned in this open issue on etcd-cluster-operator),

If you have a v1beta1 CRD that set spec.preserveUnknownFields to true (or omitted it and was defaulted to true), modifying it by explicitly setting spec.preserveUnknownFields to false is the correct way to indicate you do not want to allow arbitrary fields, and to enable publishing the openapi schema.

however I find the solution troubling, as PreserveUnknownFields is deprecated and the alternative field does not support the value False.

In apiextensions.k8s.io/v1, spec.preserveUnknownFields can only be set to false in new CRDs, and defaults to false if unspecified, so if you create a new CRD via the v1 API you'll get the behavior you want. If you have an existing CRD, you can explicitly set it to false.

Is there an expected version where PreserveUnknownFields will be removed entirely?

No.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

RamLavi · 2020-10-22T07:35:12Z

@liggitt thank you for your explained response :)

fedebongio · 2020-11-03T21:52:22Z

/triage accepted

RamLavi added the kind/bug Categorizes issue or PR as related to a bug. label Oct 19, 2020

k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 19, 2020

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 19, 2020

RamLavi mentioned this issue Oct 19, 2020

make it possible to run kubectl explain crd kubevirt/cluster-network-addons-operator#524

Merged

RamLavi changed the title ~~kubectl explain not updated when updating CRD apiVersion to apiextensions.k8s.io/v1~~ kubectl explain DESCRIPTION not updated when updating CRD apiVersion to apiextensions.k8s.io/v1 Oct 21, 2020

RamLavi changed the title ~~kubectl explain DESCRIPTION not updated when updating CRD apiVersion to apiextensions.k8s.io/v1~~ upgrades between CRDs are broken - kubectl explain not updated Oct 21, 2020

k8s-ci-robot closed this as completed Oct 21, 2020

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 3, 2020

gparvin mentioned this issue Mar 29, 2021

remove the preserveUnknownFields entry in the crd stolostron/hub-crds#26

Merged

Starefossen mentioned this issue May 10, 2021

[kong] add missing root object type to CRD OpenAPI v3 schema Kong/charts#358

Closed

3 tasks

liggitt mentioned this issue Aug 22, 2021

CRD: Top level x-kubernetes-preserve-unknown-fields breaks kubectl explain #104419

Closed

kaushik229 mentioned this issue Oct 17, 2022

🐛 Handle deprecated preserveUnknownField in CRD spec kubernetes-sigs/cluster-api-provider-nested#321

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrades between CRDs are broken - kubectl explain not updated #95702

upgrades between CRDs are broken - kubectl explain not updated #95702

RamLavi commented Oct 19, 2020 •

edited

Loading

RamLavi commented Oct 19, 2020

RamLavi commented Oct 21, 2020

liggitt commented Oct 21, 2020

k8s-ci-robot commented Oct 21, 2020

RamLavi commented Oct 22, 2020

fedebongio commented Nov 3, 2020

upgrades between CRDs are broken - kubectl explain not updated #95702

upgrades between CRDs are broken - kubectl explain not updated #95702

Comments

RamLavi commented Oct 19, 2020 • edited Loading

RamLavi commented Oct 19, 2020

RamLavi commented Oct 21, 2020

liggitt commented Oct 21, 2020

k8s-ci-robot commented Oct 21, 2020

RamLavi commented Oct 22, 2020

fedebongio commented Nov 3, 2020

RamLavi commented Oct 19, 2020 •

edited

Loading