From 410438bea752aae1de8db1bfd60ed4777b249c96 Mon Sep 17 00:00:00 2001 From: clux Date: Mon, 11 Dec 2023 04:26:11 +0000 Subject: [PATCH 1/2] Doc on admission control and more modern options - mentions our example in kube plus other users doing it manually - mentions CEL validation and roughly how to do it (linking to the new issue) - mentions CEL policies and admission policy choices Signed-off-by: clux --- docs/controllers/admission.md | 90 ++++++++++++++++++++++++++++++++++- includes/abbreviations.md | 1 + includes/links.md | 4 ++ mkdocs.yml | 1 + 4 files changed, 94 insertions(+), 2 deletions(-) diff --git a/docs/controllers/admission.md b/docs/controllers/admission.md index 443ae74..80d2644 100644 --- a/docs/controllers/admission.md +++ b/docs/controllers/admission.md @@ -1,3 +1,89 @@ -# Admission WIP +# Admission -[admission into Kubernetes](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/). +This chapter talks about controlling admission through imperative or declarative validation: + +- [admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) / admission controller frameworks +- [CRD validation with CEL](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules) +- [admission policies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) + +## Validation Using CEL Validation +CRDs (can be extended with validation rules written in [CEL](https://kubernetes.io/docs/reference/using-api/cel/), with canonical examples on [kubernetes.io crd validation-rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules). + +```yaml + openAPIV3Schema: + type: object + properties: + spec: + type: object + x-kubernetes-validations: + - rule: "self.minReplicas <= self.replicas" + message: "replicas should be greater than or equal to minReplicas." + - rule: "self.replicas <= self.maxReplicas" + message: "replicas should be smaller than or equal to maxReplicas." + properties: + ... + minReplicas: + type: integer + replicas: + type: integer + maxReplicas: + type: integer + required: + - minReplicas + - replicas + - maxReplicas +``` + +If your controller [[object]] is a CRD you own, then this is the recommended way to include validation because it is much less error prone than writing an admission controller. The feature is GA on new clusters, but otherwise generally available as __Beta__ unless your [cluster is EOL](https://endoflife.date/kubernetes): + +!!! note "Feature: CustomResourceValidationExpressions" + + This requires __Kubernetes >=1.25__ (where the feature is Beta), or Kubernetes >= 1.29 (where the [feature is GA](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md)). + + +To include validation rules in the schemas you must add `x-kubernetes-validations` entries by [[schemas#overriding-members]] for the necessary types manually (or inject them more manually). We [hope this can be made more ergonomic with future improvements to the overall ecosystem](https://github.com/kube-rs/kube/issues/1367). Help is welcome. + + +To write CEL expressions consider using the [CEL playground](https://playcel.undistro.io/). + +## Validation Using Webhooks +AKA writing an [admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/). + +These controllers __run as webservers__ rather than the traditional controller loop, and are given an [AdmissionReview] containing an [AdmissionRequest] that you must inspect and decide whether to [deny](https://docs.rs/kube/latest/kube/core/admission/struct.AdmissionResponse.html#method.deny) or accept. + +An admission controller can be [Validating](https://docs.rs/k8s-openapi/latest/k8s_openapi/api/admissionregistration/v1/struct.ValidatingWebhook.html), [Mutating](https://docs.rs/k8s-openapi/latest/k8s_openapi/api/admissionregistration/v1/struct.MutatingWebhook.html) or both. + +See the [kube::core::admission] module for how to set this up, or the [example mutating admission_controller using warp](https://github.com/kube-rs/kube/blob/main/examples/admission_controller.rs). + +!!! warning "Admission controller management is hard" + + Creating an admission webhook requires a [non-trivial amount of certificate management](https://github.com/kube-rs/kube/blob/main/examples/admission_setup.sh) for the [webhookconfiguration](https://github.com/kube-rs/kube/blob/main/examples/admission_controller.yaml.tpl), and come with its fair share of footguns (see e.g. [Benefits and Dangers of Admission Controllers KubeCon'23](https://www.youtube.com/watch?v=6kK9otYAYac)). + + Consider CEL validation / CEL policies before writing an admission controllers. + +Two examples of admission controllers in rust using kube: + +- [kuberwarden-controller](https://github.com/kubewarden/kubewarden-controller) +- [linkerd-policy-controller](https://github.com/linkerd/linkerd2/tree/main/policy-controller) + +## Validation Using Policies +External or native objects (that you do not wish to validate at the CRD level), can be validated externally using a `ValidatingAdmissionPolicy`. + +These [AdmissionPolicies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) let you inline CEL validation rules in an object similar to the WebhookConfiguration object for admission controllers, and tell Kubernetes to reject/accept for you based on simpler CEL expressions. + +!!! note "Feature: ValidatingAdmissionPolicy" + + This feature is available in __Beta in 1.28__. The talk [Declarative Everything at KubeCon'23](https://www.youtube.com/watch?v=rFaWmd7Y7i0) shows the current status of the feature and its plans for mutation. + + +## Validation Using Frameworks +If your use-case is company-wide security policies, then rather than writing an admission controller, or waiting for `AdmissionPolicy` to handle your case, consider the currently available major tooling for admission policies: + +- [Kyverno](https://kyverno.io/) - [policy list](https://kyverno.io/policies/) +- [Kubewarden](https://www.kubewarden.io/) - [oci installable policies](https://artifacthub.io/packages/search?kind=13&sort=relevance&page=1) +- [OPA Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/) - [regu policies](https://open-policy-agent.github.io/gatekeeper-library/website/) + +If you are creating validation for a CRD on the other hand, then it's less ideal to tie the validation to a particular framework as this can limit adoption of your operator. + +--8<-- "includes/abbreviations.md" +--8<-- "includes/links.md" diff --git a/includes/abbreviations.md b/includes/abbreviations.md index 8b3e1f1..822e3bf 100644 --- a/includes/abbreviations.md +++ b/includes/abbreviations.md @@ -8,3 +8,4 @@ *[GHA]: GitHub Actions *[SBOM]: Software Bill of Materials *[LTS]: Long Term Support +*[CEL]: Common Expression Language diff --git a/includes/links.md b/includes/links.md index 39b7f36..6de33ae 100644 --- a/includes/links.md +++ b/includes/links.md @@ -96,3 +96,7 @@ [k3d]: https://k3d.io/ [JsonSchema]: https://docs.rs/schemars/latest/schemars/trait.JsonSchema.html [env_logger]: https://docs.rs/env_logger/latest/env_logger/ +[AdmissionReview]: https://docs.rs/kube/latest/kube/core/admission/struct.AdmissionReview.html +[AdmissionResponse]: https://docs.rs/kube/latest/kube/core/admission/struct.AdmissionResponse.html +[AdmissionRequest]: https://docs.rs/kube/latest/kube/core/admission/struct.AdmissionRequest.html +[kube::core::admission]: https://docs.rs/kube/latest/kube/core/admission/index.html diff --git a/mkdocs.yml b/mkdocs.yml index a16ca2f..566ab2a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -76,6 +76,7 @@ nav: - controllers/manifests.md - controllers/observability.md - controllers/optimization.md + - controllers/admission.md - controllers/security.md - controllers/streams.md - controllers/generics.md From cb7abbd7dcacc07f0515e1204e9504328ef1876e Mon Sep 17 00:00:00 2001 From: clux Date: Thu, 14 Dec 2023 04:29:30 +0000 Subject: [PATCH 2/2] cross-link to validation and have a more complete crd validation example Signed-off-by: clux --- docs/controllers/admission.md | 22 ++++++++++++++++++++-- docs/controllers/schemas.md | 6 ++---- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/controllers/admission.md b/docs/controllers/admission.md index 80d2644..97c7ef7 100644 --- a/docs/controllers/admission.md +++ b/docs/controllers/admission.md @@ -41,10 +41,28 @@ If your controller [[object]] is a CRD you own, then this is the recommended way This requires __Kubernetes >=1.25__ (where the feature is Beta), or Kubernetes >= 1.29 (where the [feature is GA](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md)). -To include validation rules in the schemas you must add `x-kubernetes-validations` entries by [[schemas#overriding-members]] for the necessary types manually (or inject them more manually). We [hope this can be made more ergonomic with future improvements to the overall ecosystem](https://github.com/kube-rs/kube/issues/1367). Help is welcome. +To include validation rules in the schemas you must add `x-kubernetes-validations` entries by [[schemas#overriding-members]] for the necessary types manually (or inject them more manually): + +```rust +fn string_legality(_: &mut schemars::gen::SchemaGenerator) -> schemars::schema::Schema { + serde_json::from_value(serde_json::json!({ + "type": "string", + "x-kubernetes-validations": [{ + "rule": "self != 'illegal'", + "message": "string cannot be illegal" + }] + })) + .unwrap() +} +``` + +and this can be attached with a `#[schemars(schema_with = "string_legality)]` field attribute on some `Option` (here). See [#1372](https://github.com/kube-rs/kube/pull/1372/files) too see interactions with errors and a larger struct and other validations. + +!!! note "Future work" + Currently overriding the schema to include `x-kubernetes-validations` is awkward on larger types since you have override `items` (say) and other properties `schemars` usually generates. We hope [this can be made more ergonomic with future improvements to the overall ecosystem](https://github.com/kube-rs/kube/issues/1367). Help is welcome. -To write CEL expressions consider using the [CEL playground](https://playcel.undistro.io/). +To write CEL expressions consider using the [CEL playground](https://playcel.undistro.io/). There are more examples in the [CRD Validation Rules announcement blog](https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/) and under [kubernetes.io crd validation-rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules). ## Validation Using Webhooks AKA writing an [admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/). diff --git a/docs/controllers/schemas.md b/docs/controllers/schemas.md index dddbbf9..2f43d8b 100644 --- a/docs/controllers/schemas.md +++ b/docs/controllers/schemas.md @@ -129,9 +129,7 @@ You can define multiple structs within versioned modules ala https://github.com/ See [CustomResource#versioning](https://docs.rs/kube/latest/kube/derive.CustomResource.html#versioning), and upstream docs on [Versions in CustomResourceDefinitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) for more info. ## Validation -Kubernetes >1.25 supports including [validation rules in the openapi schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules), and there are a couple of ways to include these. - -See the [Openapi V3 blogpost for more context](https://kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga/). +Kubernetes >1.25 supports including [validation rules in the openapi schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules), and there are a couple of ways to include these. See [[admission#Validation Using CEL Validation]] for examples. ### Manual Rules This can be done by following upstream docs, and manually [[#Implementing-JsonSchema]] or [[#Overriding-Members]] to inject validation rules into specific parts of the schema. @@ -152,4 +150,4 @@ See [CustomResource#schema-validation](https://docs.rs/kube/latest/kube/derive.C [//begin]: # "Autogenerated link references for markdown compatibility" [#overriding]: schemas "Schemas" -[//end]: # "Autogenerated link references" \ No newline at end of file +[//end]: # "Autogenerated link references"