Kubernetes upgrade notes: 1.19.x to 1.20.x

If you used my Kubernetes the Not So Hard Way With Ansible blog posts to setup a Kubernetes (K8s) cluster this notes might be helpful for you (and maybe for others too that manage a K8s cluster on their own on prem e.g.).

I’ve a general upgrade guide Kubernetes the Not So Hard Way With Ansible - Upgrading Kubernetes that worked quite well for me for the last past K8s upgrades. So please read that guide if you want to know HOW the components are updated. This post here is esp. for the 1.19.x to 1.20.x upgrade and WHAT was interesting for me.

First: As usual I don’t update a production system before the .2 release of a new major version is released. In my experience the .0 and .1 are just too buggy (and to be honest sometimes it’s even better to wait for the .5 release ;-) ). Of course it is important to test new releases already in development or integration systems and report bugs!

Second: I only upgrade from the latest version of the former major release. In my case I was running 1.19.4 and at the time writing this text 1.19.12 was the latest 1.19.x release. After reading the 1.19.x changelog to see if any important changes where made between 1.19.4 and 1.19.12 I don’t saw anything that prevented me updating and I don’t needed to change anything. So I did the 1.19.4 to 1.19.12 upgrade first. If you use my Ansible roles that basically only means to change k8s_release variable from 1.19.4 to 1.19.12 and roll the changes for the control plane and worker nodes out as described in my upgrade guide. After that everything still worked as expected so I continued with the next step.

Here are a few links that might be interesting regarding what’s new in regards to new features in Kubernetes 1.20:

Kubernetes CHANGELOG
Kubernetes 1.20: The Raddest Release
What’s new in Kubernetes 1.20 - SysDig blog

Since K8s 1.14 there are also searchable release notes available. You can specify the K8s version and a K8s area/component (e.g. kublet, apiserver, …) and immediately get an overview what changed in that regard. Quite nice! :-)

As it is normally no problem to have a newer kubectl utility that is only one major version ahead of the server version I also updated kubectl from 1.19.x to 1.20.8 using my kubectl Ansible role.

As always before a major upgrade read the Urgent Upgrade Notes! If you used my Ansible roles to install Kubernetes and used most of the default settings then there should be no need to adjust any settings.

There was at least one change that required changes to my Ansible K8s controller role:

• TokenRequest and TokenRequestProjection are now GA features. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object.

This feature requires three flags:

• --service-account-key-file, set to one or more files containing one or more public keys used to verify tokens. That one I already added in a previous release.
• --service-account-issuer, should be set to a URL identifying the API server that will be stable over the cluster lifetime. My role sets this flag to the fully qualified domain name (FQDN) of the first controller node (e.g. controller01.i.domain.tld). If you’ve a load balancer in front of your kube-apiserver daemons that URL might also be a good candidate.
• --service-account-signing-key-file, set to a file containing a private key to use to sign service account tokens. Can be the same file given to kube-controller-manager with --service-account-private-key-file.

Further interesting notes:

• Dockershim deprecation: Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. The Kubernetes community has written a blog post about this in detail with a dedicated FAQ page for it.
• PID Limits graduates to General Availability: PID Limits features are now generally available on both SupportNodePidsLimit (node-to-pod PID isolation) and SupportPodPidsLimit (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year.
• IPv4/IPv6 run: IPv4/IPv6 dual-stack has been re-implemented for 1.20 to support dual-stack Services, based on user and community feedback. Currently in alpha. It’s expected that dual-stack implementation to progress from alpha to beta and GA in coming releases. This is a BREAKING CHANGE to an alpha API.
• Introducing Graceful Node Shutdown (Alpha): While still in alpha that’s an interesting feature. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown.
• kubectl debug graduates to Beta
• Introducing RootCAConfigMap - RootCAConfigMap graduates to Beta, separating from BoundServiceAccountTokenVolume. The kube-root-ca.crt ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections.
• Pod Hostname as FQDN graduates to Beta - Previously introduced in 1.19 behind a feature gate, SetHostnameAsFQDN is now enabled by default. More details on this behavior are available in documentation for DNS for Services and Pods
• The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags --address and --insecure-bind-address have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags --port and --insecure-port may only be set to 0 and will be removed in v1.24.
• TokenRequest and TokenRequestProjection are now GA features. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object.

If you use CSI then also check the CSI Sidecar Containers documentation. Every sidecar container contains a matrix which version you need at a minimum, maximum and which version is recommend to use with whatever K8s version. Nevertheless if your K8s update to v1.20 worked fine I would recommend to also update the CSI sidecar containers sooner or later because a) lots of changes happen ATM in this area and b) you might require the newer versions for the next K8s version anyways.

Now I finally updated the K8s controller and worker nodes to version 1.20.8 as described in Kubernetes the Not So Hard Way With Ansible - Upgrading Kubernetes.

If you see errors like

kube-controller-manager[3375]: E0405 18:58:30.109867    3375 leaderelection.go:331]
error retrieving resource lock kube-system/kube-controller-manager:
leases.coordination.k8s.io "kube-controller-manager" is forbidden:
User "system:kube-controller-manager" cannot get resource "leases" in API group
"coordination.k8s.io" in the namespace "kube-system"

during upgrading the controller nodes then this seems to be ok. The error should go away if all controller nodes are using the new Kubernetes version (also see 1879).

That’s it for today! Happy upgrading! ;-)