v2.25.1

Changes by Kind

Deprecation / Removal

• Deprecating support for Centos7 and Weave ; they are not tested anymore (#11428, @k8s-infra-cherrypick-robot)
• Remove Debian 10 support. (#11429, @k8s-infra-cherrypick-robot)ra-cherrypick-robot)

Feature

• Default to Kubernetes v1.29.10 (#11666, @enneitex)
• Add hashes for Kubernetes v1.29.9 (#11599, @robertvolkmann), Add hashes for Kubernetes v1.29.7 (#11408, @mzaian), Add hashes for Kubernetes v1.29.6 (#11355, @tico88612)
• Default to etcd v3.5.16 (#11576, @janosbabik)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11464, @k8s-infra-cherrypick-robot)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11249, @k8s-infra-cherrypick-robot)
• CentOS 7 yum repo baseurl update (#11365, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11438, @k8s-infra-cherrypick-robot)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11339, @k8s-infra-cherrypick-robot)

Applications

• [helm] Upgrade to v3.14.4 (#11599, @robertvolkmann)

Network

• [calico] Update default calico to v3.27.4
  [calico] Fix high cpu load due to XDP program in iptables (#11476, @mzaian)

Container-Managers

• [containerd] Default to v1.7.22
  [nerdctl] Upgrade to 1.7.7
  [runc] Upgrade to v1.1.14 (#11576, @janosbabik)

Bug or Regression

• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11616, @k8s-infra-cherrypick-robot)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11332, @k8s-infra-cherrypick-robot)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11638, @k8s-infra-cherrypick-robot)

v2.24.3

Changes by Kind

API Change

• Default to kubernetes v1.28.14
  Default to etcd v3.5.16
  Default to containerd v1.7.22
  Default to cri-o v1.28.10
  Default to nerdctl 1.7.7
  Default to runc v1.1.14 (#11516, @VannTen)

Feature

• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11361, @k8s-infra-cherrypick-robot)
• Check CentOS-Base.repo exists for CentOS 7 (#11546, @tico88612)

Other (Cleanup or Flake)

• Update KUBESPRAY_VERSION in galaxy.yml to v2.24.3 and Update Readme.md to v2.24.1 (#11385, @yankay)

v2.26.0

Deprecation / Removal

• Deprecating support for Centos7; they are not tested anymore (#11344, @ant31)
• Remove Debian 10 support. (#11347, @tico88612)
• Remove the kubeadm_version which is always equal to kube_version (#11473, @VannTen)
• Drop support for Kubernetes 1.27.x minimum version now is 1.28.x (#11221, @mzaian)
• if you were previously only setting serializeImagePulls: false to have unlimited parallel pulls, you will need to set kubelet_max_parallel_images_pulls to a suitable value instead (#11094, @tu1h)

Feature / Major Changes

• Make kubernetes v1.30.4 default (#11455, @kokyhm)
• Add hashes for Kubernetes v1.30.3 default (#11391, @tico88612), Add hashes for Kubernetes v1.30.2 default (#11343, @tmurakam), Add hashes for Kubernetes 1.30.0, 1.30.1 and 1.30.2 (#11261, @tmurakam), Add hashes for kubernetes 1.29.7, 1.28.[11-12] (#11407, @mzaian)
• Add option ubuntu_kernel_unattended_upgrades_disabled to control unattended-upgrades for Linux kernel and all packages start with linux- on Ubuntu (#11296, @tu1h)
• Added option to configure dependencies for kubelet.service (#11297, @ledroide)
• Adds the possibility to add extra arguments to the various containers in the cinder-csi plugin.(#11169, @Payback159)
• Allow to run kubespray with an empty kube_node group, to provision only the control plane (#11248, @VannTen)
• CentOS 7 yum repo baseurl update (#11360, @tico88612)
• Check CentOS-Base.repo exists for CentOS 7 (#11402, @tu1h)
• Check if peers is defined when peering with routers (#11259, @ehsan310)
• OpenStack Cloud Controller Manager upgrade to 1.30.0 (#11358, @tico88612)
• Rename systemd module to systemd_service (#11396, @tu1h)
• User has the ability to configure calico-kube-controllers log level (#11335, @mirwan)
• User has the ability to configure local_volume_provisioner log level (#11336, @mirwan)
• User has the ability to configure netchecker components log levels (#11334, @mirwan)
• You can now disable installing OS dependencies using system's package manager by skipping system-packages tag. (#10872, @hedayat)
• kubelet_max_parallel_image_pulls represents the maximum number of image pulls in parallel (#11094, @tu1h)
• Update reset task to support Tencent OS (reset_restart_network_service_name) (#11459, @KubeKyrie)
• Add conditional checking on ubuntu kernel unattended_upgrades disabling (#11479, @tu1h)

Applications

• Bump Cinder CSI Plugin to v1.30.0 (#11374, @tico88612)
• Bump upcloud csi driver to v1.1.0 in order to enable csi volume snapshots. (#11303, @Elias-elastisys)
• User has a possibility to fix nodePort of ingress-nginx service with property in addons.yaml (#11310, @mochizuki875)
• Update kube-vip to v0.8.0 (#11156, @jisnardo)
• [cert-manager] upgrade to v1.14.7 (#11341, @tico88612)
• [cert-manager] add support v1.13.6 (#11279, @tico88612)
• [ingress-nginx] upgrade controller to version 1.11.2 (#11463, @mzaian)
• [helm] Upgrade to v3.15.4, add 3.15.x, and drop 3.13.x (#11486, @yankay)
• Add support for LB in UpCloud private zone (#11260, @davidumea)
• Bump UpCloud terraform module to v5.6.0
  UpCloud servers specify server groups to be apart of, eliminates manual rescheduling. (#11311, @robinAwallace)
• Update node-feature-discovery to v0.16.4 (#11250, @mzaian)
• Allow for configuring etcd progress notify interval and default set to 5s (#11499, @liuxu623)
• Support Gateway API CRDs install (#11376, @tico88612)
• Increase ansible timeout to 300 (#11354, @rptaylor)

Network

• [calico] Change calico default version to v3.28.1, add v3.28.0 and checksum , Update calico apiserver deployment to use new readiness probe (#11234, @ehsan310)
• [calico] add calico support v3.27.4 to fix high cpu load due to XDP program in iptables (#11476, @ehsan310)
• Add cilium_hubble_event_buffer_capacity & cilium_hubble_event_queue_size vars (#10943, @pedro-peter)
• [network] bump cni version to v1.4.0 (#10698, @cyclinder)
• Change weave CNI to community version and upgrade to the latest version (2.8.7) (#11228, @tico88612)
• [kube-ovn] update to v1.12.21 (#11445, @oilbeater)

Container-Managers

• [containerd] Make containerd 1.7.21 default (#11478, @yankay)
• [containerd] added debug config variables (#11080, @spnngl)
• [containerd] fixes wrong templating for tracing config (#11372, @ugur99)
  [runc] Upgrade to v1.1.13 (#11413, @mzaian)
• Update docker cli version 26.1.2 (#11291, @ErikJiang)

Documentation

• Update RELEASE.md for upgrading k8s (#11321, @yankay)

Bug or Regression

• Delete /etc/NetworkManager/conf.d/dns.conf on reset. (#11440, @HoKim98)
• Fix Hetzner kubernetes group names (#11232, @jmaccabee13)
• Fix: skip multus when not defined (#10934, @darkobas2)
• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11309, @mochizuki875)
• Provide missing advertise-address flag to kube-apiserver (#11387, @derselbst)
• Update reset task to support Kylin OS (reset_restart_network_service_name) (#11406, @KubeKyrie)
• Updated indentation in cni-kube-ovn.yml.j2 (L658) (#11357, @sanshah1211)
• Fix CI with fail docker pull in gitlab runner by change DOCKER_HOST (#11315, @yankay)
• Fix etcd not starting up when using a custom access address (#11388, @derselbst)
• Fix the Auto Bump PR is blocked by the label do-not-merge/release-note-label-needed by adding dependabot release-note-none label. (#11256, @yankay)
• Fix kube_reserved so it only controls kubeReservedCgroup . (#11367, @rptaylor)
• Disables reconfiguring the cluster during upgrade (remove --config option from kubeadm upgrade apply) (#11352, @tmurakam)
• Fix error in boostrap-os when git does not handle symlinks (#11508, @VannTen)
• Fix static kube-apiserver advertise address based on first control plane (#11457, @Seljuke)
• Fix incorrect member matching when removing etcd nodes (#11488, @ErikJiang)
• Fix double pop of access_ip (#11435, @rptaylor)
• Fix use super-admin.conf for kube-vip on first master when it exists to support initial k8s v1.29+ installation with kube-vip enabled (#11422, @Seljuke)

Other (Cleanup or Flake)

• Contrib playbooks are no longer included in the ansible kubespray collection (#11239, @VannTen)
• Reduced required python packages in requirements.txt (#11199, @itayporezky)
• Fix openstack cleanup by change the delete security_group order (#11299, @yankay)
• RHEL 7, Centos 7 and derivatives are no longer supported. (#11246, @VannTen)
• Use TasksMask=infinity on ostree systems for docker systemd service (#11493, @VannTen)

Supported Components

• Core
  • kubernetes v1.30.4
  • etcd v3.5.12
  • docker v26.1
  • containerd v1.7.21
  • cri-o v1.30.3 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.28.1
  • cilium v1.15.4
  • flannel v0.22.0
  • kube-ovn v1.12.21
  • kube-router v2.0.0
  • multus v3.8
  • weave v2.8.7
  • kube-vip v0.8.0
• Application
  • cert-manager v1.14.7
  • coredns v1.11.1
  • ingress-nginx v1.11.2
  • krew v0.4.4
  • argocd v2.11.0
  • helm v3.15.4
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.30.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0
  • node-feature-discovery v0.16.4

Known issues

N/A

Notes

• Deprecating support for Centos7
• The Ansible version has been upgrade to 9.8.0
• Change weave CNI to community version https://github.com/weaveworks/weave

Maintainers

Great respect for joining maintainers 🎉

• Adding myself (VannTen) as approver #11483 @VannTen
• Add tico88612 as reviewer #11453 @tico88612
• owners: move ant31 from emeritus_approvers to approvers #11247 @ant31

v2.24.2

Changes by Kind

Feature

• Make kubernetes v1.28.10 default (#11269, @mzaian)
• Revert 'Support CoreDNS use host network & config CoreDNS port' (#10617, @liuxu623)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (#11330, @mochizuki875)

Bug or Regression

• Ingress-nginx-controller admission service is automatically created when ingress_nginx_webhook_enabled: true (#11331, @mochizuki875)
• Fix CentOS 7 yum repo baseurl update (#11364, @tico88612 )

Other (Cleanup or Flake)

• Remove the archived debian apt repository when installing docker-engine (#11215, @VannTen )
• Update KUBESPRAY_VERSION in galaxy.yml v2.24.1 (#10961, @yankay)

v2.25.0

Deprecation / Removal

• Remove support for Kubernetes 1.26.x (move min version to 1.27.x) (#10817, @KubeKyrie)
• Remove documentation for removed in-tree openstack provider (#10889, @LarssonOliver)

Feature / Major Changes

• A check is introduced to fail the playbook if cgroups are not enabled on the node (#11165, @franznemeth)
• Add Calico v3.27.3 and make it default (#11141, @pomland-94)
• Add extra_vars support to vagrant setup (#10932, @VannTen)
• Add kube-vip LeaderElection variables vip_leaseduration, vip_renewdeadline, vip_retryperiod options for kube-vip (#11021, @KubeKyrie)
• Add new option remove_anonymous_access to prevent granting RBAC permissions to anonymous users. (#11016, @nicolas-goudry)
• Add scheduler plugins support (scheduler_plugins_enabled enable or disable the installation scheduler plugins / scheduler_plugins_enabled_plugins describe the enabled plugins / scheduler_plugins_diabled_plugins describe the disabled plugins / scheduler_plugins_plugin_config set the custom config for enabled plugins) (#10747, @tu1h)
• Added a config option to filter ntp interfaces (#11066, @Pavan-Gunda)
• Adding egress IPv6 for node-local-dns queries (k8s_allowed_egress_ipv6_ips) (#10396, @raviranjanelastisys)
• Bump docker version for kylin linux (#11203, @ErikJiang)
• Bump docker version for openeuler linux (#11206, @ErikJiang)
• Update almalinux-8 base image to 8.9 (#10918, @VannTen)
• Bumping checksums and various versions (#10999, @MrFreezeex)
• Containerd: allow to configure fallback server (#10988, @sathieu)
• Docker upgrade from 24.0 to 26.1 (#11198, @tico88612)
• Download hash script: auto discover versions (#10849, @VannTen)
• Enable configuring mountOptions, reclaimPolicy and volumeBindingMode for cinder-csi StorageClasses. (#10450, @Payback159)
• Make containerd v1.7.15 default (#11083, @Payback159)
• Make kubernetes v1.28.6 default (#10810, @mzaian)
• Make kubernetes v1.29.1 default
  Remove SecCompDefault feature gate from hardening configuration for kubernetes 1.29 (#10820, @tmurakam)
• Make kubernetes v1.29.2 default (#10919, @mzaian)
• Make kubernetes v1.29.3 default (#11035, @mzaian)
• Make kubernetes v1.29.4 default (#11108, @mzaian)
• Make kubernetes v1.29.5 default (#11196, @mzaian)
• Metallb: added metallb_namespace variable to parameterize namespace (#11136, @oik741)
• OpenStack Cloud Controller Manager upgrade to 1.28.2 (#11174, @tico88612)
• Opensuse deployment is now tested in CI. (#11159, @VannTen)
• Add selinux-ng repo in Amazon Linux to install container-selinux (#11182, @yankay)
• Add CI Image for Ubuntu 24.04 (#11167, @yankay)
• Allows .vagrant folder location to be configured (#10718, @kri5)
• Prevent nodelocaldns to be OOM-killed (#11056, @sathieu)
• Support Node Feature Discovery (#10861, @yankay)
• Support Ubuntu 24.04 (#11132, @tico88612)
• Support following k8s version selection pause image (#10756, @my-git9)
• The variable old_dns_domains (list) can be used for backward compatibility when changing dns_domain (#10630, @VannTen)
• Update external huawei cloud controller to 0.26.6 (#10824, @dabeck)
• Update external huawei cloud controller to 0.26.8 (#11172, @dabeck)
• Update kube-vip to v0.8.0 (#11156, @jisnardo)
• Update metrics server to v0.7.0 (#10856, @mzaian)
• Updated ingress controller version to 1.9.6 (#10868, @kundan2707)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (#10925, @chrxmvtik)
• [Terraform-openstack] Added possibility to build an octavia loadbalancer for the Kubernetes Api. (#10924, @jaszil)
• [containerd] added distributed tracing config variables for containerd (containerd_tracing_enabled,containerd_tracing_endpoint,containerd_tracing_protocol, containerd_tracing_sampling_ratio,containerd_tracing_service_name ); it is disabled by default. (#11103, @ugur99)
• [etcd] Default version to 3.5.12 for k8s 1.27 , 1.28 , 1.29 (#11036, @mzaian)
• Minimum ansible-core version is now 2.16.4 (#10984, @VannTen)
• Remove the archived debian apt repository when installing docker-engine (#11088, @yankay)
• Change dependbot interval to weekly (#11189, @yankay)
• Allow specifying CPU Manager Policy options through kubelet_cpu_manager_policy_options (#11023, @derselbst)
• [kube-apiserver] added distributed tracing config variables for kube-apiserver (kube_apiserver_tracing,kube_apiserver_tracing_endpoint,kube_apiserver_tracing_sampling_rate_per_million); it is disabled by default.
  [kubelet] added distributed tracing config variables for kubelet (kubelet_tracing,kubelet_tracing_endpoint,kubelet_tracing_sampling_rate_per_million); it is disabled by default. (#10795, @ugur99)

Applications

• [argocd] update argocd to v2.11.0 (#11193, @mzaian)
• [helm] Upgrade to v3.14.2 (#10967, @cleman95)
• Bump coredns version to 1.11.1 (#10719, @batazor)
• Support CoreDNS use host network & config CoreDNS port (#10617, @liuxu623)
• Fix secondary coredns missing var (#10821, @VannTen)
• Revert "support CoreDNS use host network and config dns port (#10617)" (#11185, @VannTen)
• dns_mode: coredns_dual is now tested in CI. (#10903, @VannTen)

Network

• Adds support for cilium v1.15
  • Adds support for cilium_l2announcements to replace metallb with cilium l2 announcements, defaults to false
  • Adds support for cilium_loadbalancer_mode to switch bpf-lb-mode between snat, dsr or hybrid, default to snat (#11106, @deveshk0)
• Adds the option to install calico 3.27.3 (#11059, @danielfrg)
• [calico] Update default calico to v3.27.2 (#10960, @mzaian)

Container-Managers

• crictl stop container grace period, cri_stop_containers_grace_period: 0 (#10651, @krembu)
• Update the docker default version to 24.0 (#10873, @yankay)
• [Containerd] Enable by default discard_unpacked_layers to save some space (see containerd/containerd#6295) (#10905, @VannTen)
• [Nerdctl] Upgrade to version 1.7.4 (#10968, @cleman95)
• [containerd] Make containerd 1.7.13 default
  [runc] Upgrade to v1.1.12 (#10862, @KubeKyrie)
• [containerd] Make containerd 1.7.16 default (#11142, @mzaian)

API Change

• Make proxy protocol in Upcloud LB configurable (#10971, @davidumea)

Design

• Merge stop and remove systemd service task in reset/tasks/main.yml (#10902, @kimsehwan96)

Documentation

• Add documentation for configuring nat outgoing ipv6 (#10866, @anders-elastisys)
• Add new OpenStack Cloud for terraform (#10910, @DragomirAlin)
• BREAKING CHANGE: This script is introduced to facilitate living documentation and its administration. This leads to a restructuring in the documentation at https://kubespray.io/#/ to simplify the automatic creation of links, as the structure in the sidebar changes. (#11128, @Payback159)
• Change a task name Ensure kube-bench parameters are set into Ensure kubelet expected parameters are set in roles/kubernetes/preinstall/tasks/0080-system-configurations.yml for a clearer understanding of its operation (#11171, @kimsehwan96)
• Do not disable SELinux surreptitiously (#10920, @rptaylor)
• Doc clarification: skipping patches releases is OK (#10850, @VannTen)
• Docs: vagrant-libvirt is tested in CI (#10847, @VannTen)
• Explicit private/public nature of *ip vars (#10904, @VannTen)
• Fix typo in vagrant.md (#10836, @kundan2707)
• Fix typo mistake in roles/kubernetes/control-plane/tasks/define-first-kube-control.yml (#10835, @kimsehwan96)
• Fixed typos in inventory/sample/group_vars/k8s_cluster (#10911, @arahmangulov)
• Kubespray used as a collection will have the correct collection version (#10727, @VannTen)
• Make large-deployments.md link to downloads.md (#10840, @spantaleev)
• Removed not needed graduated feature gates. (#10448, @Smidra)
• Update upgrades.md with serial=1 for rolling updates (#10837, @titansmc)
• Variable cilium_ipsec_key must be base64 encoded (#10781, @ledroide)

Bug or Regression

• Added an optional variable (cni_bin_owner) to allow the user to set a different owner for /opt/cni/bin/ and it's contents. (#10929, @Rickkwa)
• Change the position of the containerd_extra_args parameter to enhance its universality. (#11013, @qcu266)
• Configure crio container runtime to use kube reserved cgroup (#11028, @pedro-peter)
• Don't overwrite changes to openstack allowed_address_pairs #10760 (#10760, @rptaylor)
• Download cache directory permissions are no longer reset recursively (#10900, @VannTen)
• Fix ClusterRole for Calico >=v1.26.x with Calico API Server installed (#11089, @RaSerge)
• Fix ansible parameter ssh_args in ansible.cfg file not work (#10981, @joy717)
• Fix boostrap for Amazon Linux (#11139, @VannTen)
• Fix crio registries config file when using slashes in the registry path (#11030, @pedro-peter)
• Fix file loss during download (#10779, @ErikJiang)
• Fix kubespray-defaults: Check for boostrap-os FQCN (#11073, @KubeKyrie)
• Fix local path provisioner image repo in sample inventory. (#11180, @tico88612)
• Fix logical error when checking for boostrap-os (#10867, @VannTen)
• Fix lsattr command error when kubelet has symbolic link (#11074, @KubeKyrie)
• Fix network manage service of Debian 12 (#11058, @KubeKyrie)
• Fix nginx controller leader election RBAC (#10913, @VannTen)
• Fix python regex matching problem when finding docker packages (#11075, @KubeKyrie)
• Fix waiting for MetalLB controller (#10858, @flxbwr)
• Fix(kubernetes): taint nodes on cluster upgrade (#10705, @maxime1907)
• Fix: config hostname as string type in kubeadmConfig rendering (#10997, @ErikJiang)
• Fixes running recover-control-plane.yml with offline broken etcd nodes. (#10660, @yuha0)
• Revert OCCM standard dnsPolicy to ClusterFirst to fix #10914 which was introduced with #10618 and make dnsPolicy configurable to furthermore support #10618 (#11168, @Payback159)
• Force update helm repo if ...

v2.24.1

Changes by Kind

Feature

• Make kubernetes v1.28.6 default (#10810, @mzaian)

Bug or Regression

• Add configuration to create cilium CNI plugin file when cilium>=1.14.0 (#10945, @cleman95 )
• Fix logical error when checking for boostrap-os (#10953, @VannTen)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Patch GHSA-xr7r-f8xq-vfvv (#10877, @VannTen)

Other (Cleanup or Flake)

• Bump galaxy version before release (#10890, @VannTen)

---

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.22.2

Changes by Kind

Network

• [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)

API Change

• Make kubernetes 1.26.11 default (#10704, @VannTen)

Feature

• Add hashes for kubernetes version 1.26.6, 1.26.7, 1.26.8 & 1.26.9 (#10444, @bozzo)
• Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
• Make kubernetes 1.26.13 the default version (#10823, @VannTen)

Failing Test

• Bump vagrant version 2.3.7 (#10789, @yankay)

Bug or Regression

• Fix hardcoded pod infra version (#10805, @ErikJiang)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Patch GHSA-xr7r-f8xq-vfvv (#10878, @VannTen)
• [Multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

---

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.23.3

Changes by Kind

Feature

• Update kubernetes default version to 1.27.10 (#10876, @VannTen)

Bug or Regression

• Fix hardcoded pod infra version (#10806, @ErikJiang)
• Make containerd 1.7.13 default
  Make runc 1.1.12 default
  Make kubernetes 1.27.10 default
  Patch GHSA-xr7r-f8xq-vfvv (#10876, @VannTen)

Other (Cleanup or Flake)

• Update KUBESPRAY_VERSION in galaxy.yml and Readme for v2.23.2 (#10801, @yankay)

---

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.24.0

Deprecation / Removal

• Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (#10464, @unai-ttxu)
• Drop support for Kubernetes 1.25.x (move min version to 1.26.x) (#10420, @yankay)
• Drop installation notes for Debian Jessie (#10642, @jelmer)

Feature / Major Changes

• Make kubernetes v1.28.6 default (#10810, @mzaian)
• Add kubernetes v1.28.0, v1.28.1, v1.28.2, v1.28.3, v1.28.4, v1.28.5 hash (#10435, #10541, #10739, @mzaian ; #10390, @tmurakam ; #10624, @tmurakam)
• Add Retry for Applying PriorityClass (#10469, @hangscer8)
• Add option crio_criu_support_enabled to enable container forensic analysis (#10479, @tu1h)
• Add option kubectl_alias to set bash alias of kubectl (#10552, @tu1h)
• Add variable to configure ipvs modules (kube_proxy_ipvs_modules) (#10580, @borgiacis)
• Check nameserver only when dns is enable (#10561, @yckaolalala)
• Correctly handle remove_default_searchdomains when value is undefined (#10533, @yckaolalala)
• Kube-scheduler: remove/update deprecated component component config v1beta3. (#10484, @mzaian)
• Terraform-aws: variable driven ami selection (ami_name_pattern/ami_virtualization_type/ami_owners) (#10520, @mertcancam)
• Terraform-openstack: Added possibility to enable dhcp flag critical on one interface (#10446, @Xartos)
• This will introduce a new variable kube_apiserver_admission_plugins_podnodeselector_default_node_selector that can be used with kube_apiserver_admission_plugins_needs_configuration: [PodNodeSelector] defined. So allows the users to configure PodNodeSelector plugin. (#10607, @titansmc)
• UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from anti_affinity_policy to anti_affinity) (#10474, @robinAwallace)
• Update dockerfile to follow best practices (#10708, @maxime1907)
• Update to ansible 2.15 and set minimum version to 2.15.5 (#10481, @MrFreezeex)
• [etcd] Update Default etcd version to 3.5.10 for kubernetes 1.28, 1.27 and 1.26 (#10798, @VannTen)
• [etcd] update version to 3.5.9 for k8s 1.28 , 1.27 , 1.26 (#10482, @mzaian)
• [etcd] add 3.5.10 hashes (#10566, @mzaian)
• [vsphere_csi] Update to 3.1.0 supports Kubernetes Version 1.28 (#10451, @mzaian)
• [cinder_csi] Cinder-CSI now use cluster_name variable instead of the default hardcoded "kubernetes" value (#10422, @floryut)

Applications

• [argocd] update argocd to v2.8.4 (#10568, @mzaian)
• [helm] upgrade to 3.13.1 (#10567, @mzaian)
• [coredns] Added option coredns_additional_error_config to allow for configuration of the coredns error plugin. (#10501, @Elias-elastisys)
• [coredns] Support CoreDNS use host network & config CoreDNS port (#10617, @liuxu623)
• [coredns] Support disable dns autoscaler when use CoreDNS (#10608, @liuxu623)
• [coredns] Add pdb to coredns (#10557, @lobiyedKarim1)
• [cert-manager] upgrade to v1.13.2 (#10616, @liuxu623)
• [cert-manager] Upgrade to v1.12.6 (#10582, @chansuke)
• [cert-manager] Upgrade to v1.12.5 (#10500, @chansuke)

Network

• [cilium] Fix invalid hubble yaml if cilium_hubble_tls_generate is enabled (#10430, @toonalbers)
• [cilium] Use correct ports in cilium metrics services if metrics are enabled. (#10519, @bakito)
• [cilium] Adds support for deploying clusters with cilium 1.14+ (#10684, @rl0nergan)
• [calico] Separate calico-node and calico-cni-plugin service accounts and update default calico to v3.26.1 (#10416, @mzaian)
• [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)
• [calico] Update default calico to v3.26.3 (#10526, @mzaian)
• [calico] Update default calico to v3.26.4 (#10669, @mzaian)
• [kube-router] Default kube-router version updated to v2.0.0 (#10503, @bozzo)
• [kube-router] Default kube-router version updated to v1.6.0 (#10478, @bozzo)
• [kube-router] Add kube_router_bgp_graceful_restart optional setting for disabling graceful BGP restarts (default to true) (#10489, @rosskusler)
• [metallb] Add option to set avoidBuggyIPs in IPAddressPools and change the default back to false (#10458, @zeeZ)
• [metallb] Metallb --lb-class cmd arg to support multiple LoadBalancer implementations (#10550, @Seal1998)
• [custom_cni] Add helm support for custom_cni deployment (#10529, @kukacz)
• [kube_vip] Add kube_vip_lb_fwdmethod option for kube-vip (#10762, @tu1h)

Container-Managers

• [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)
• [containerd] Make containerd 1.7.11 default (#10671, @mzaian)
• [containerd] Add hashes for containerd versions 1.7.6 ~ 1.7.8 default (#10439, #10525, #10589, @mzaian)
• [containerd] Specify the runc path when we use the containerd container engine and change the bin_dir path. (#10154, @qlijin)
• [containerd] Refactor NRI activation for containerd and CRI-O (remove crio_enable_nri and containerd_nri_disable) now only one var nri_enabled default to false (#10470, @fmuyassarov)
• [containerd] Add Boolean option enable_cdi to enable cdi (false by default) (#10603, @krembu)
• [containerd] Add configuration option for NRI (disable by default) in crio & containerd (using new containerd_nri_disable and crio_enable_nri) (#10454, @fmuyassarov)
• [containerd] add config support override_path (#10776, @yankay)
• [runc] Upgrade to v1.1.10 (#10671, @mzaian)
• [crio] Update to v1.28.1 (#10480, @qlijin)
• [crio] Remove crio package configuration during cleanup (#10584, @yckaolalala)
• [crio] Update docs for crio_registry_auth (#10785, @qlijin)
• [docker] Ability to define GPG key path for Docker APT (using new variable docker_repo_key_keyring) (#10513, @emiran-orange)
• [kata-containers] Freshens configuration-qemu to latest template compatible with kata-containers 3.1.3. (#10466, @Alphadelta14)
• [nerdctl] Bump nerdctl version 1.7.1 (#10685, @yankay)
• [nerdctl] Change nerdctl version from 1.5.0 to 1.6.0 (#10475, @MaGaroo)

Documentation

• Add link to Cilium CNI documentation (#10431, @toonalbers)
• Update docs for calico_iptables_backend in Redhat/Centos.md (#10417, @yankay)
• Update metallb example configs (#10485, @caruccio)
• Updated AWS ALB ingress controller version (#10680, @kundan2707)

Bug or Regression

• Add a variable reset_restart_network_service_name in the reset role to be able to configure the name of the service which is restarted. (#10428, @RomainMou)
• Add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true (#10618, @Payback159)
• Check for correct conntrack module presence, regardless of kernel versions (#10662, @VannTen)
• Fallback_ips: ignore unreachable hosts (#10601, @poblahblahblah)
• Fix 'kube-apiserver' tag inappropriately overwriting secrets at rest encryption token (#10460, @jwitko)
• Fix assertion for task item verify-settings (#10699, @piwinkler)
• Fix external-lb in kubelet.conf server address and kube-proxy api-server address (#10490, @ugur99)
• Fix forgotten update of etcd-servers list in apiserver manifest when scaling (#8253, @liupeng0518)
• Fix metallb example yaml (#10545, @caruccio)
• Fix reset job for cri-o container engine (#10197, @turbosnail)
• Fix restart network task cannot be skipped (ansible boolean conversion needed) (#10512, @ErikJiang)
• Fix: add kubelet tag in task of Fetch facts to avoid kubelet config inconsistencies (#10423, @NierYYDS)
• Fixes the path of the certificates use in the etcdctl.sh wrapper when the deployment type is not kubeadm. (#10467, @RomainMou)
• Hubble relay will work when cilium_cluster_name is customised. (#10614, @eugene-eeo)
• Disable podCIDR allocation from control-plane when using calico (#10639, @VannTen)
• Kubespray-defaults: Check for boostrap-os FQDN (#10590, @VannTen)
• Patch for modprobe_nf_conntrack for new Linux Kernel, when using ipvs (#10625, @abhishekkr)
• Remove always tag applied on bootstrap (#10556, @yckaolalala)
• Set remove_default_searchdomains to false by default (#10554, @hedayat)
• Swap is now disabled using systemd (mask of swap.target) (#10587, @VannTen)
• Fix undefined retries variable when copying etcdctl (#10634, @ErikJiang)
• Move control plane certs renewal "spread out" into the systemd timer (#10596, @VannTen)
• The dhcp configuration for dns nameservers are now the same than during installation (#10548, @smutel)
• Use correct env var name for kube-vip per service leader election (#10433, @ThisIsQasim)
• Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
• Fix download retry when get_url has no status_code (#10613, @RomainMou)
• Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
• Set the maxUnavailable of the coredns rolling update strategy to 1 (#10748, @tu1h)
• Fix crio_version version comparison (#10780, @ledroide)
• Fix disable swap failed in Centos/RHEL 7 (#10751, @yankay)
• Fix image pull fail with insecure-registry (#10775, @yankay)
• Refactor check_galaxy + fix version (#10729, @VannTen)
• Fix Helm installation on SLES and openSUSE (#10794, @goldyfruit)
• Fix incorrect ciliumcli binary (#10575, @tu1h)
• Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
• Fix the cluster installation on cluster using etcd clients nodes (cilium / calico / ...) (#10769, @VannTen)

Other (Cleanup or Flake)

• Cleanup a deprecation warning (ipaddr filter) (#10518, @VannTen)
• Decouple kubespray-defaults from download (#10626, @VannTen)
• Etcd/backup: use native ansible modules instead of shell (#10540, @VannTen)
• Etcd: use dynamic group for certs generation check (#10610, @VannTen)
• Factorize some identical playbooks steps into their own sub-playbooks (#10633, @VannTen)
• Pre-upgrade tasks cleanup (#10656, @VannTen)
• Refactor "multi" handlers to use listen (#10542, @VannTen)
• Remove unneeded workaround for removing kubeadm DNS (#10695, @VannTen)
• Removed DEPRECATED --logtostderr from metrics-server (#10709, @micha...

v2.23.2

Container-Managers

• [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)

API Change

• Make kubernetes 1.27.9 the default version (#10797, @VannTen)

Feature

• Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
• Update kubernetes default version to 1.27.9
• Update etcd version for 1.27 and 1.26 to 3.5.10 (#10797, @VannTen)

Failing Test

• Bump vagrant version 2.3.7 (#10788, @yankay)

Bug or Regression

• Fix calico-node in etcd mode. (#10768, @VannTen)
• Fix download retry when get_url has no status_code (#10613, @RomainMou) (#10791, @VannTen)
• Kube-controller-manager will no longer assign pod CIDRs to cluster nodes when using calico (with its default IPAM, calico_ipam_host_local now has a default value of false) [⚠️ NOTE users using a non-true value for calico_ipam_host_local will need to change it to true] (#10639, @VannTen)

Other (Cleanup or Flake)

• Kubespray collection will have the correct collection version. (#10728, @VannTen)