Kubernetes upgrade notes: 1.34.x to 1.35.x

Introduction
If you used my Kubernetes the Not So Hard Way With Ansible blog posts to setup a Kubernetes (K8s) cluster this notes might be helpful for you (and maybe for others too that manage a K8s cluster on their own e.g.). I’ll only mention changes that might be relevant because they will either be interesting for most K8s administrators anyways (even in case they run a fully managed Kubernetes deployment) or if it’s relevant if you manage your own bare-metal/VM based on-prem Kubernetes deployment. I normally skip changes that are only relevant for GKE, AWS EKS, Azure or other cloud providers.
I’ve a general upgrade guide Kubernetes the Not So Hard Way With Ansible - Upgrading Kubernetes that worked quite well for me for the last past K8s upgrades. So please read that guide if you want to know HOW the components are updated. This post here is esp. for the 1.34.x to 1.35.x upgrade and WHAT was interesting for me.
As usual I normally don’t update a production system before the .2 release of a new major version is released. In my experience the .0 and .1 are just too buggy (well, it got way better meanwhile but you don’t want do experiments in production, right? 😉). Nevertheless it’s important to test new releases (and even beta or release candidates if possible) already in development environments and report bugs!
Update to latest current release
I only upgrade from the latest version of the former major release. At the time writing this blog post 1.34.8 was the latest 1.34.x release. After reading the 1.34 CHANGELOG to figure out if any important changes where made between the current 1.34.x and latest 1.34.8 release I didn’t see anything that prevented me updating and I don’t needed to change anything.
So I did the 1.34.8 update first. If you use my Ansible roles that basically only means to change k8s_ctl_release variable from 1.34.x to 1.34.8 (for the controller nodes) and the same for k8s_worker_release (for the worker nodes). Deploy the changes for the control plane and worker nodes as described in my upgrade guide.
Hint: To save some time, IMHO it should be good enough to only update the controller nodes to the latest 1.34.x release as it’s mostly the kube-apiserver that stores the state of the Kubernetes cluster in etcd and that state is quite important. That’s what I normally do. Upgrading to the next major release can then be done for all nodes as usual. But if you want to be absolutely sure just upgrade the whole cluster to the latest 1.34.x release first.
After that everything still worked as expected, I continued with the next step.
Upgrading kubectl
As it’s normally no problem (and actually the supported method) to have a newer kubectl utility that is only one major version ahead of the server version I updated kubectl from 1.32.x to latest 1.33.x using my kubectl Ansible role.
Upgrading etcd
While my roles are not using kubeadm to manage my K8s cluster, it’s recommended to have at least etcd 3.5.24 running but 3.5.26 is definitely the better option (see below). I updated my etcd role to current 3.5.26 and updated my etcd deployment accordingly. See Upgrading Kubernetes - etcd for more information how to upgrade etcd.
NOTE 1: There is already etcd 3.6.x available. But for K8s 1.35 I’ll stay with etcd 3.5 for now. etcd 3.6.x has some breaking changes that needs to be addressed first. Upgrading etcd to 3.6.x is planned with the next major release upgrade.
NOTE 2: If you use etcd for Kubernetes and want to upgrade to etcd v3.6 you DEFINITELY want to upgrade to etcd v3.5.26 first! For further information:
Upgrading containerd
containerd was updated from 2.2.1 to 2.3.0. I updated my containerd role accordingly. Please read the CHANGELOG for potential breaking changes. From my experience the upgrade “just works”. But if you upgrade that role before version 0.17.0+2.2.1 then please check the CHANGELOG.
containerd >= 2.0 is also a prerequisite to use user namespace isolation which got enabled by default as of Kubernetes 1.33.
Upgrading runc
runc was upgraded from 1.3.2 to 1.4.2. I’ve updated my runc role accordingly. Release notes for runc are in runc releases but shouldn’t be that interesting.
With runc >= 1.2 and containerd >= 2.0 (as mentioned above) User Namespaces are finally supported. Together with Kubernetes release > v1.33 user namespace isolation finally arrived out of the box and you should consider using it.
Upgrading CNI
And finally the CNI plugins are updated from 1.8.0 to 1.9.1. Again I updated my CNI role accordingly. The release notes for CNI 1.9.1 might be worth a read but only if you want to go deeper 😉
Release notes
Since K8s 1.14 there are also searchable release notes available. You can specify the K8s version and a K8s area/component (e.g. kubelet, apiserver, …) and immediately get an overview what changed in that regard. Quite nice! 😉
Urgent Upgrade Notes
NOTE: If you use etcd for Kubernetes and want to upgrade to etcd v3.6 you DEFINITELY want to upgrade to etcd v3.5.26 first! For further information:
I guess most users wont be affected by any Urgent Upgrade Notes. Here are the potentially important ones:
- Removed the
--pod-infra-container-imageflag from kubelet command line. cgroups v1wont work out of the box anymore.kubeletwont start ifcgroups v1is detected. This most probably only affects users running Kubernetes on older Linux distributions that do not supportcgroups v1.
What’s New (Major Themes)
All important stuff is listed in the Kubernetes v1.35: Timbernetes release announcement.
The following list of changes and features only contains stuff that I found useful and interesting. That means I’m normally not mentioning any Kubernetes internals that have changed but mostly stuff that is interesting for administrators and operation. This is mainly to remember myself what changed 😉 See the full Kubernetes v1.35 Changelog for all changes.
Graduated to stable
Graduated to beta
Alpha features
Deprecations
- cgroup v1 support as mentioned above.
- Deprecation of ipvs mode in kube-proxy. Currently it’s only deprecated. It’s currently the default for service load balancing in my kubernetes_worker Ansible role. With the next major release of that role it will be replaced by
nftables. Another possibility would be to disablekube-proxyand use Cilium kube-proxy Replacement. - Kubernetes is deprecating containerd v1.y support: This is the last call to switch to containerd
v2.x. My githubixx.containerd already supportsv2.3. If you still runv1.7upgrade tov2.3shouldn’t be too complicated. Also see githubixx.containerd changelog.
API changes
Features
Other
etcd: Updated tov3.5.26
Further reading
- Kubernetes v1.35 Sneak Peek
- Kubernetes 1.35 - New security features
- Kubernetes 1.35 Timbernetes Introduces Vertical Scaling
- Kubernetes as AI’s operating system: 1.35 release signals
- Kubernetes v1.35: Job Managed By Goes GA
- Kubernetes 1.35: In-Place Pod Resize Graduates to Stable
- Avoiding Zombie Cluster Members When Upgrading to etcd v3.6
- Kubernetes v1.35: Kubelet Configuration Drop-in Directory Graduates to GA
- Kubernetes v1.35: Fine-grained Supplemental Groups Control Graduates to GA
- Kubernetes v1.35: Introducing Workload Aware Scheduling
- Kubernetes v1.35: Watch Based Route Reconciliation in the Cloud Controller Manager
- Kubernetes 1.35: Enhanced Debugging with Versioned z-pages APIs
- Kubernetes v1.35: New level of efficiency with in-place Pod restart
- Kubernetes v1.35: Extended Toleration Operators to Support Numeric Comparisons (Alpha)
- Kubernetes v1.35: A Better Way to Pass Service Account Tokens to CSI Drivers
- Kubernetes v1.35: Mutable PersistentVolume Node Affinity (alpha)
- Kubernetes v1.35: Restricting executables invoked by kubeconfigs via exec plugin allowList added to kuberc
CSI
If you use CSI then also check the CSI Sidecar Containers documentation. Every sidecar container contains a matrix which version you need at a minimum, maximum and which version is recommend to use with whatever K8s version.
Nevertheless if your K8s update to v1.35 worked fine I would recommend to also update the CSI sidecar containers sooner or later.
Upgrade Kubernetes
Now I finally upgraded the K8s controller and worker nodes to version 1.35.x as described in Kubernetes the Not So Hard Way With Ansible - Upgrading Kubernetes.
That’s it for today! Happy upgrading! 😉