diff --git a/docs/controllers/admission.md b/docs/controllers/admission.md new file mode 100644 index 0000000..443ae74 --- /dev/null +++ b/docs/controllers/admission.md @@ -0,0 +1,3 @@ +# Admission WIP + +[admission into Kubernetes](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/). diff --git a/docs/controllers/application.md b/docs/controllers/application.md new file mode 100644 index 0000000..a8955fd --- /dev/null +++ b/docs/controllers/application.md @@ -0,0 +1,11 @@ +# Application WIP + +The controller **application** watches **object** changes and forwards the main **object** to the reconciler. + +This doc is a **WIP**. + +plan: + +- standard build setup (cargo + ci) +- main controller glue (Controller invocation, linking to further topics) +- containerising (with musl or distroless) diff --git a/docs/controllers/intro.md b/docs/controllers/intro.md new file mode 100644 index 0000000..4321e15 --- /dev/null +++ b/docs/controllers/intro.md @@ -0,0 +1,126 @@ +# Introduction + +This guide showcases how to build controllers with kube-rs, and is a WIP (2 chapters done). + +## Overview + +A controller a long-running program that ensures the kubernetes state of an object, matches the state of the world. + +As users update the desired state, the controller sees the change and schedules a reconciliation, which will update the state of the world: + +```mermaid +flowchart TD + A[User/CD] -- kubectl apply object.yaml --> K[Kubernetes Api] + C[Controller] -- watch objects --> K + C -- schedule object --> R[Reconciler] + R -- result --> C + R -- update state --> K +``` + +any unsuccessful reconciliations are retried or requeued, so a controller should **eventually** apply the desired state to the world. + +Writing a controller requires **three** pieces: + +- an **object** dictating what the world should see +- an **reconciler** function that ensures the state of one object is applied to the world +- an **application** living in kubernetes watching the object and related objects + +## The Object + +The main object is the source of truth for what the world should be like, and it takes the form of a Kubernetes object like a: + +- [Pod](https://arnavion.github.io/k8s-openapi/v0.14.x/k8s_openapi/api/core/v1/struct.Pod.html) +- [Deployment](https://arnavion.github.io/k8s-openapi/v0.14.x/k8s_openapi/api/apps/v1/struct.Deployment.html) +- ..[any native Kubernetes Resource](https://arnavion.github.io/k8s-openapi/v0.14.x/k8s_openapi/trait.Resource.html#implementors) +- a partially typed or dynamically typed Kubernetes Resource +- an object from [api discovery](https://docs.rs/kube/latest/kube/discovery/index.html) +- a [Custom Resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) + +Kubernetes already has a [core controller manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) for the core native objects, so the most common use-case for controller writing is a **Custom Resource**, but many more fine-grained use-cases exist. + +See the [[object]] document for how to use the various types. + +## The Reconciler + +The reconconciler is the part of the controller that ensures the world is up to date. + +It takes the form of an `async fn` taking the object along with some context, and performs the alignment between the state of world and the `object`. + +In its simplest form, this is what a reconciler (that does nothing) looks like: + +```rust +async fn reconcile(object: Arc, data: Context) -> + Result +{ + // TODO: logic here + Ok(ReconcilerAction { + requeue_after: Some(Duration::from_secs(3600 / 2)), + }) +} +``` + +As a controller writer, your job is to complete the logic that align the world with what is inside the `object`. +The core reconciler must at **minimum** contain **mutating api calls** to what your `object` is meant to manage, and in some situations, handle annotations management for [ownership](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/) or [garbage collection](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/). + +Writing a goood **idempotent reconciler** is the most difficult part of the whole affair, and its difficulty is the reason we generally provide diagnostics and observability: + +See the [[reconciler]] document for a run-down with all the best-practices. + +## The Application + +The controller application is the part that watches for changes, determines what root object needs reconciliations, and then schedules reconciliations for those changes. It is the glue that turns what you want into __something__ running in Kubernetes. + +In this guide; the **application** is written in [rust], using the [kube] crate as a **dependency** with the `runtime` feature, compiled into a **container**, and deployed in Kubernetes as a **`Deployment`**. + +The core features inside the application are: + +- an encoding of the main object + relevant objects +- an infinite [watch loop](https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes) around relevant objects +- a system that maps object changes to the relevant main object +- an **idempotent reconciler** acting on a main object + +The system must be **fault-tolerant**, and thus must be able to recover from **crashes**, **downtime**, and resuming even having **missed messages**. + +Setting up a blank controller in rust satisfying these constraints is fairly simple, and can be done with minimal boilerplate (no generated files need be inlined in your project). + +See the [[application]] document for the high-level details. + +## Controllers and Operators + +The terminology between **controllers** and **operators** are quite similar: + +1. Kubernetes uses the following [controller terminology](https://kubernetes.io/docs/concepts/architecture/controller/): + +> In Kubernetes, controllers are **control loops** that watch the **state** of your cluster, then make or request **changes where needed**. Each controller tries to move the current cluster state closer to the desired state. + +2. The term **operator**, on the other hand, was originally introduced by `CoreOS` as: + +> An Operator is an application-specific controller that extends the Kubernetes API to create, configure and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts, but also includes domain or application-specific knowledge to automate common tasks better managed by computers. + +Which is further reworded now under their new [agglomerate banner](https://cloud.redhat.com/learn/topics/operators). + +They key **differences** between the two is that **operators** generally a specific type of controller, sometimes more than one in a single application. A controller would at the very least need to: + +- manage custom resource definition(s) +- maintain single app focus + +to be classified as an operator. + +The term **operator** is a flashier term that makes the **common use-case** for user-written CRD controllers more understandable. If you have a CRD, you likely want to write a controller for it ([otherwise why](https://kubernetes.io/docs/concepts/configuration/configmap/) go through the effort of making a custom resource?). + +## Guide Focus + +Our goal is that with this guide, you will learn how to use and apply the various controller patterns, so that you can avoid scaffolding out a large / complex / underutilized structure. + +We will focus on all the patterns as to not betray the versatility of the Kubernetes API, because components found within complex controllers can generally be mixed and matched as you see fit. + +We will focus on how the variour element **composes** so you can take advantage of any controller archetypes - operators included. + +--8<-- "includes/abbreviations.md" +--8<-- "includes/links.md" + +[//begin]: # "Autogenerated link references for markdown compatibility" +[object]: object "The Object" +[reconciler]: reconciler "Reconciler WIP" +[application]: application "Application WIP" +[//end]: # "Autogenerated link references" diff --git a/docs/controllers/object.md b/docs/controllers/object.md new file mode 100644 index 0000000..5e11b87 --- /dev/null +++ b/docs/controllers/object.md @@ -0,0 +1,249 @@ +# The Object + +A controller always needs a __source of truth__ for **what the world should look like**, and this object **always lives inside kubernetes**. + +Depending on how the object was created/imported or performance optimization reasons, you can pick one of the following object archetypes: + +- typed Kubernetes native resource +- Derived Custom Resource for Kubernetes +- Imported Custom Resource already in Kubernetes +- untyped Kubernetes resource +- partially typed Kubernetes resource + +We will outline how they interact with controllers and the basics of how to set them up. + +## Typed Resource + +This is the most common, and simplest case. Your source of truth is an existing [Kubernetes object found in the openapi spec](https://arnavion.github.io/k8s-openapi/v0.14.x/k8s_openapi/trait.Resource.html#implementors). + +To use a typed Kubernetes resource as a source of truth in a [Controller], import it from [k8s-openapi], and create an [Api] from it, then pass it to the [Controller]. + +```rust +use k8s_openapi::api::core::v1::Pod; + +let pods = Api::::all(client); +Controller::new(pods, ListParams::default()) +``` + +This is the simplest flow and works right out of the box because the openapi implementation ensures we have all the api information via the [Resource] traits. + +If you have a native Kubernetes type, **you generally want to start with [k8s-openapi]**. If will likely do exactly what you want without further issues. **That said**, if both your clusters and your chosen object are large, then you can **consider optimizing** further by changing to a [partially typed resource](#partially-typed-resource) for smaller memory profile. + +A separate [k8s-pb] repository for our [future protobuf serialization structs](https://github.com/kube-rs/kube-rs/issues/725) also exists, and while it will slot into this category and should hotswappable with [k8s-openapi], it is **not yet usable** here. + +## Custom Resources +### Derived Custom Resource + +The operator use case is heavily based on you writing your own struct, and a schema, and [extending the kuberntes api](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) with it. + +This **has** historically required a lot of boilerplate for both the api information and the (now required) [schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules), but this is a lot simpler with kube thanks to the [CustomResource] derive [proc_macro]. + +```rust +/// Our Document custom resource spec +#[derive(CustomResource, Deserialize, Serialize, Clone, Debug, JsonSchema)] +#[kube(kind = "Document", group = "kube.rs", version = "v1", namespaced)] +#[kube(status = "DocumentStatus")] +pub struct DocumentSpec { + name: String, + author: String, +} + +#[derive(Deserialize, Serialize, Clone, Debug, JsonSchema)] +pub struct DocumentStatus { + checksum: String, + last_updated: Option>, +} +``` + +This will generate a `pub struct Document` in this scope which implements [Resource]. In other words, to use it with the a controller is at this point analogous to a fully typed resource: + +```rs +let docs = Api::::all(client); +Controller::new(pods, ListParams::default()) +``` + +!!! note "Custom resources require schemas" + + Since **v1** of [CustomResourceDefinition] became the main variant ([`v1beta1` was removed in Kubernetes 1.22](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#removal-of-several-beta-kubernetes-apis)), a schema is **required**. These schemas are generated using [schemars] by specifying the `JsonSchema` derive. See the schemas section (TODO) for further information on advanced usage. + +#### Installation + +Before Kubernetes accepts api calls for a custom resource, we need to install it. This is the usual pattern for creating the yaml definition: + +```toml +# Cargo.toml +[[bin]] +name = "crdgen" +path = "src/crdgen.rs" +``` + +```rust +// crdgen.rs +use kube::CustomResourceExt; +fn main() { + print!("{}", serde_yaml::to_string(&mylib::Document::crd()).unwrap()) +} +``` + +Here, a separate `crdgen` bin entry would install your custom resource using `cargo run --bin crdgen | kubectl -f -`. + +!!! warning "Installation outside the controller" + + While it is tempting to install a custom resource within your controller at startup, this is not advisable. The permissions needed to write to the cluster-level `customresourcedefinition` resource is almost always much higher than what your controller needs to run. It is thus advisable to generate the yaml out-of-band, and bundle it with the rest of the controller's installation yaml. + +### Imported Custom Resource + +In the case that a `customresourcedefinition` **already exists** in your cluster, but it was **implemented in another language**, then we can **generate structs from the schema** using [kopium]. + +Suppose you want to write some extra controller or replace the native controller for `PrometheusRule`: + +```sh +kopium prometheusrules.monitoring.coreos.com --docs > prometheusrule.rs +``` + +this will read the crd from the cluster, and generate rust-optimized structs for it: + +```rust +use kube::CustomResource; +use serde::{Serialize, Deserialize}; +use std::collections::BTreeMap; + +/// Specification of desired alerting rule definitions for Prometheus. +#[derive(CustomResource, Serialize, Deserialize, Clone, Debug)] +#[kube(group = "monitoring.coreos.com", version = "v1", kind = "PrometheusRule", plural = "prometheusrules")] +#[kube(namespaced)] +#[kube(schema = "disabled")] +pub struct PrometheusRuleSpec { + /// Content of Prometheus rule file + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub groups: Vec, +} + +/// RuleGroup is a list of sequentially evaluated recording and alerting rules. Note: PartialResponseStrategy is only used by ThanosRuler and will be ignored by Prometheus instances. Valid values for this field are 'warn' or 'abort'. More info: https://github.com/thanos-io/thanos/blob/master/docs/components/rule.md#partial-response +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct PrometheusRuleGroups { + pub interval: Option, + pub name: String, + pub partial_response_strategy: Option, + pub rules: Vec, +} + +/// Rule describes an alerting or recording rule See Prometheus documentation: [alerting](https://www.prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) or [recording](https://www.prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) rule +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct PrometheusRuleGroupsRules { + pub alert: Option, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub annotations: BTreeMap, + pub expr: String, + pub r#for: Option, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub labels: BTreeMap, + pub record: Option, +} +``` + +you typically would then import this file as a module and use it as follows: + +```rust +use prometheusrule::PrometheusRule; + +let rules: Api = Api::default_namespaced(client); +Controller::new(rules, ListParams::default()) +``` + +!!! warning "Kopium is unstable" + + Kopium is a relatively new project and it is [neither feature complete nor bug free at the moment](https://github.com/kube-rs/kopium/issues). While feedback has been very positive, and people have so far contributed fixes for several major customresources; **expect some snags**. +## Dynamic Typing + + +### Untyped Resources + +Untyped resources are using [DynamicObject]; an umbrella container for arbitrary Kubernetes resources. + +!!! warning "Hard to use with controllers" + + This type is the most unergonomic variant available. You will have to operate on [untyped json](https://docs.serde.rs/serde_json/#operating-on-untyped-json-values) to grab data out of specifications and is best suited for general (non-controller) cases where you need to look at common metadata properties from [ObjectMeta] like `labels` and `annotations` across different object types. + +The [DynamicObject] consists of **just the unavoidable properties** like `apiVersion`, `kind`, and `metadata`, whereas the entire spec is loaded onto an arbitrary [serde_json::Value] via [flattening]. + +The benefits you get is that: + +- you avoid having to write out fields manually +- you **can** achieve tolerance against multiple versions of your object +- it is compatible with api [discovery] + +but you do have to find out where the object lives on the api (its [ApiResource]) manually: + +```rust +use kube::{api::{Api, DynamicObject}, discovery}; + +// Discover most stable version variant of `documents.kube.rs` +let apigroup = discovery::group(&client, "kube.rs").await?; +let (ar, caps) = apigroup.recommended_kind("Document").unwrap(); + +// Use the discovered kind in an Api, and Controller with the ApiResource as its DynamicType +let api: Api = Api::all_with(client, &ar); +Controller::new_with(api, ListParams::default(), &ar) +``` + +Other ways of doing [discovery] are also available. We are highlighting [recommended_kind] in particular here because it can be used to achieve version agnosticity. + +!!! note "Multiple versions of an object" + + Kubernetes supports specifying [multiple versions of a specification](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/), and using [DynamicObject] above can help solve that. There are [other potential ways](https://github.com/kube-rs/kube-rs/issues/569) of achieving similar results, but it does require some work. + +### Partially-typed Resource + +These resources specify a subset of the normal typed information, and allow you to improve memory characterstics of the program, at the cost of slightly more `struct` code. + +It is similar to [DynamicObject] (above) in that [Object] is another umbrella container for arbitrary Kubernetes resources, and also requires you to discover or hard-code an [ApiResource] for extra type information to be queriable. + +Here is an example of handwriting a new implementation of [Pod] by overriding its **spec** and **status** and placing it inside [Object], then **stealing** its type information: + +```rust +use kube::api::{Api, ApiResource, NotUsed, Object}; + +// Here we replace heavy type k8s_openapi::api::core::v1::PodSpec with +#[derive(Clone, Deserialize, Debug)] +struct PodSpecSimple { + containers: Vec, +} +#[derive(Clone, Deserialize, Debug)] +struct ContainerSimple { + #[allow(dead_code)] + image: String, +} +// Pod replacement +type PodSimple = Object; + +// steal api resource information from k8s-openapi +let ar = ApiResource::erase::(&()); + +Controller::new_with(api, ListParams::default(), &ar) +``` + +In the end, we end up with some extra lines to define our [Pod], but we also drop every field inside spec + status except `spec.container.image`. If your cluster has thousands of pods and you want to do some kind of common operation on a small subset of fields, then this can give a very quick win in terms of memory use (a Controller will usually maintain a `Store` of all owned objects). + +### Dynamic new_with constructors + +!!! warning "Partial or dynamic typing always needs additional type information" + + All usage of `DynamicObject` or `Object` require the use of alternate constructors for multiple interfaces such as [Api] and [Controller]. These constructors have an additional `_with` suffix to carry an associated type for the [Resource] trait. + +## Summary + +All the fully typed methods all have a **consistent usage pattern** once the types have been generated. The dynamic and partial objects have more niche use cases and require a little more work such as alternate constructors. + +| typing | Source | Implementation | +| ------------------------- | ------------------------------------ |---------------------------- | +| :material-check-all: full | [k8s-openapi] | `use k8s-openapi::X` | +| :material-check-all: full | kube::[CustomResource] | `#[derive(CustomResource)]` | +| :material-check-all: full | [kopium] | `kopium > gen.rs` | +| :material-check: partial | kube::core::[Object] | partial copy-paste | +| :material-close: none | kube::core::[DynamicObject] | write nothing | + + + +--8<-- "includes/abbreviations.md" +--8<-- "includes/links.md" diff --git a/docs/controllers/reconciler.md b/docs/controllers/reconciler.md new file mode 100644 index 0000000..72e8147 --- /dev/null +++ b/docs/controllers/reconciler.md @@ -0,0 +1,29 @@ +# Reconciler WIP + + +The reconciler is the warmest user-defined code in your controller, and it will end up doing a range of tasks. + +## Using Context + +- extracting a `Client` or an `Api` from the `Data` + + +## Idempotency + +!!! warning "A reconciler must be [idempotent](https://en.wikipedia.org/wiki/Idempotence)" + + If a reconciler is triggered twice for the same object, it should cause the same outcome. Care must be taken to not repeat expensive api calls when unnecessary, and the flow of the reconciler must be able to recover from errors occurring in a previous reconcile run. + +## OwnerReferences + +## Finalizers + +## Observability + +- tracing instrumentation of the fn +- metrics + +## Diagnostics + +- api updates to the `object`'s **status struct** +- `Event` records populated for diagnostic informatio diff --git a/includes/abbreviations.md b/includes/abbreviations.md new file mode 100644 index 0000000..a52b06b --- /dev/null +++ b/includes/abbreviations.md @@ -0,0 +1 @@ +*[HPA]: Horizontal Pod Autoscaler diff --git a/includes/links.md b/includes/links.md new file mode 100644 index 0000000..1cf9c3f --- /dev/null +++ b/includes/links.md @@ -0,0 +1,24 @@ +[Controller]: https://docs.rs/kube/latest/kube/runtime/struct.Controller.html +[Api]: https://docs.rs/kube/latest/kube/struct.Api.html +[CustomResource]: https://docs.rs/kube/latest/kube/derive.CustomResource.html +[Resource]: https://docs.rs/kube/latest/kube/trait.Resource.html +[ApiResource]: https://docs.rs/kube/latest/kube/core/struct.ApiResource.html +[Object]: https://docs.rs/kube/latest/kube/core/struct.Object.html +[DynamicObject]: https://docs.rs/kube/latest/kube/core/struct.DynamicObject.html +[discovery]: https://docs.rs/kube/latest/kube/discovery/index.html +[recommended_kind]: https://docs.rs/kube/latest/kube/discovery/struct.ApiGroup.html#method.recommended_kind +[kube]: https://crates.io/crates/kube +[kopium]: https://github.com/kube-rs/kopium +[controller-rs]: https://github.com/kube-rs/controller-rs +[version-rs]: https://github.com/kube-rs/version-rs +[ObjectMeta]: https://docs.rs/kube/latest/kube/core/struct.ObjectMeta.html +[flattening]: https://serde.rs/attr-flatten.html +[schemars]: https://graham.cool/schemars/ +[serde_json::Value]: https://docs.serde.rs/serde_json/enum.Value.html +[json]: https://docs.serde.rs/serde_json/macro.json.html +[proc_macro]: https://doc.rust-lang.org/reference/procedural-macros.html +[k8s-pb]: https://github.com/kube-rs/k8s-pb +[k8s-openapi]: https://crates.io/crates/k8s-openapi +[Pod]: https://docs.rs/k8s-openapi/latest/k8s_openapi/api/core/v1/struct.Pod.html +[CustomResourceDefinition]: https://docs.rs/k8s-openapi/latest/k8s_openapi/apiextensions_apiserver/pkg/apis/apiextensions/v1/struct.CustomResourceDefinition.html +[rust]: https://www.rust-lang.org/ diff --git a/mkdocs.yml b/mkdocs.yml index 72eb38e..70a2538 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -34,8 +34,8 @@ theme: name: Switch to dark mode language: en features: - #- navigation.tabs - - navigation.indexes + - navigation.tabs + #- navigation.indexes - navigation.sections - navigation.instant - navigation.tracking @@ -44,16 +44,32 @@ theme: # Generated from "fd .md ." in docs/ folder. # mkdocs serve will tell us about uncovered documents or broken links nav: -#- index.md -- getting-started.md -#- quick-tutorial.md -- changelog.md -#- integrations.md -- security.md -- release-process.md -- rust-version.md -- architecture.md -- adopters.md +- Information: + #- index.md + - getting-started.md + #- quick-tutorial.md + - changelog.md + #- integrations.md + - security.md + - release-process.md + - rust-version.md + - architecture.md + - adopters.md + +- Controller Guide: + - controllers/intro.md + - controllers/object.md + + #- comunity.md + #- proposal-process.md + #- mentorship.md + #- coding-style.md +- Crates: + - crates/kube.md + - crates/kube-core.md + - crates/kube-client.md + - crates/kube-derive.md + - crates/kube-runtime.md # Maybe something for the future #- Proposals/Rejected: [] @@ -67,16 +83,6 @@ nav: - website.md - tools.md - #- comunity.md - #- proposal-process.md - #- mentorship.md - #- coding-style.md -- Crates: - - crates/kube.md - - crates/kube-core.md - - crates/kube-client.md - - crates/kube-derive.md - - crates/kube-runtime.md markdown_extensions: - attr_list # https://squidfunk.github.io/mkdocs-material/reference/images/ - pymdownx.tabbed # https://squidfunk.github.io/mkdocs-material/reference/content-tabs/ @@ -90,6 +96,8 @@ markdown_extensions: guess_lang: false linenums: false - footnotes + - abbr + - pymdownx.snippets - meta - def_list - pymdownx.arithmatex @@ -108,6 +116,13 @@ markdown_extensions: - pymdownx.superfences - pymdownx.tasklist - pymdownx.tilde + # mermaid diagrams + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + plugins: - search - roamlinks diff --git a/requirements.txt b/requirements.txt index e5cfc13..9080187 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ mkdocs==1.2.3 -mkdocs-material==8.2.1 +mkdocs-material==8.2.3 mkdocs-roamlinks-plugin mkdocs-exclude diff --git a/sync.sh b/sync.sh index bad652b..92033f3 100755 --- a/sync.sh +++ b/sync.sh @@ -16,6 +16,8 @@ sync() { # Concat original file contents curl -sSL "https://raw.githubusercontent.com/${repopath}" >> "${namelocal}" # TODO: swap to use the github api tool ^ to avoid being rate limited in the future + + # TODO: fix relative links in vendored docs somehow } main() {