Kubernetes the not so hard way with Ansible - WireGuard - (K8s v1.21)
- added note about Hetzner Cloud Networks
- added comment about setting IP forwarding via
This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Provisioning Compute Resources. Since we don’t use AWS or Google Cloud we don’t have the feature set of this two platforms at hand. But we can work around the shortcoming’s with some tools and one of this tool is WireGuard.
One hint about Hetzner Cloud Networks: Just because they are private doesn’t mean that they are secure ;-) Traffic is NOT encrypted in that case. So for secure communication you still need something like WireGuard. And if you want connect your laptop at home with your servers somewhere a VPN is also quite handy ;-)
I used PeerVPN before but that wasn’t updated for a while. As I moved my cloud hosts from Scaleway to Hetzner cloud it was a good time to switch the VPN solution ;-) In general PeerVPN still works perfectly fine esp. if you need a easy to setup fully meshed network (where every node is able to talk to all other nodes and even if node
A should be able to talk to Node
C via node
B ;-) ). But PeerVPN needs also lot of CPU resources and throughput isn’t that good. That’s solved with WireGuard.
In harden the instances we secured our instances. Since we don’t have networking features like AWS VPC or Google Cloud Engine VPC we create our own secure network with WireGuard. All Kubernetes services will only listen on the WireGuard interface (
wg0 by default) for incoming requests and all communication between the Kubernetes hosts will be secured by WireGuard VPN. We’ll configure the Ansible WireGuard role to create a fully meshed VPN which includes at least all Kubernetes hosts. But you can also include additional hosts and even your laptop on which you execute Ansible commands or later even more important the Kubernetes control utility
I’ve prepared a Ansible role for installing WireGuard. You can again use
ansible-galaxy install githubixx.ansible_role_wireguard
to install the role or just clone it via git if you like.
By default port
51820 (protocol UDP) should be accessible from the outside. But you can adjust the port by changing the variable
group_vars/all.yml or in
host_vars for a specific host. Also IP forwarding needs to be enabled e.g. via
echo 1 > /proc/sys/net/ipv4/ip_forward. I decided not to implement this tasks in this Ansible role. IMHO that should be handled elsewhere. But you can use the
wireguard_(preup|predown|postup|postdown) hooks for such kind of tasks. E.g. you can define in
group_vars/all.yml) for one, some or all hosts something like this:
wireguard_preup: - echo 1 > /proc/sys/net/ipv4/ip_forward - ufw allow 51820/udp
This would enable IP forwarding and open port 51820 for UDP protocol before the WireGuard interfaces are created (in this case I assumed that UFW - Uncomplicated Firewall is used as an iptables fronted which is the case if you use harden-linux).
If you followed my blog series so far then you already know that my Ansible role harden-linux is also able to do this for you. See my previous blog post Kubernetes the not so hard way with Ansible - Harden the instances how to implement it. Besides changing sysctl entries (which you need to enable IP forwarding) it also manages firewall settings among other things. As a reminder the following settings should be set in
group_vars/all.yml (and don’t forget to roll out the changes before you roll out WireGuard ;-) ):
harden_linux_ufw_rules: - rule: "allow" to_port: "22222" protocol: "tcp" - rule: "allow" to_port: "51820" protocol: "udp" harden_linux_sysctl_settings_user: "net.ipv4.ip_forward": 1 "net.ipv6.conf.default.forwarding": 1 "net.ipv6.conf.all.forwarding": 1 harden_linux_ufw_defaults_user: "^DEFAULT_FORWARD_POLICY": 'DEFAULT_FORWARD_POLICY="ACCEPT"'
You should already have a Ansible
host_vars file for every host that is a member of your Kubernetes cluster and maybe also for your workstation you execute
kubectl later e.g.:
host_vars/controller01.i.domain.tld host_vars/controller02.i.domain.tld host_vars/controller03.i.domain.tld host_vars/worker01.i.domain.tld host_vars/worker02.i.domain.tld host_vars/workstation
In Kubernetes the not so hard way with Ansible - Harden the instances I already showed you examples what settings you should set for every host so I won’t repeat everything here again. Just the example for the first controller node:
Ansible host file:
--- wireguard_address: "10.8.0.101/24" wireguard_port: 51820 wireguard_endpoint: "controller01.p.domain.tld" wireguard_persistent_keepalive: "30" ansible_host: "controller01.p.domain.tld" ansible_port: 22222 ansible_user: dauser ansible_become: true ansible_become_method: sudo ansible_python_interpreter: /usr/bin/python3
You need to set
wireguard_address for every node. Of course every node needs a different IP but make sure to use the same network like
/24 in the example above. You also need to set
wireguard_endpoint for every node that will be part of your Kubernetes cluster. Only the workstation will have a
wireguard_endpoint set to
"" (empty string). The reason for this is that the workstation won’t have a WireGuard VPN endpoint where the other hosts would connect to. You just want to access the Kubernetes cluster from your workstation and not the other way round. But you want to enable all Kubernetes hosts to communicate with each other. That’s what
wireguard_endpoint is needed for. Besides other things the Ansible role will create a WireGuard configuration for every host. That configuration will include the information what WireGuard IP can be reached via what endpoint. You can find a complete example in the README of the WireGuard role but here is a example of the WireGuard configuration of the workstation:
[Interface] Address = 10.8.0.2/24 PrivateKey = .... ListenPort = 51820 [Peer] PrivateKey = .... AllowedIPs = 10.8.0.101/32 Endpoint = controller01.p.domain.tld:51820 [Peer] PrivateKey = .... AllowedIPs = 10.8.0.102/32 Endpoint = controller02.p.domain.tld:51820 ...
AllowedIPs parameter is also some kind of routing information and WireGuard will create routes accordingly. So on my workstation with the IP
10.8.0.2 there will be a route that tells WireGuard that if I want to connect to
10.8.0.101 (which is the first Kubernetes controller node) send the traffic to
Include the role into your playbook like in this example:
- hosts: vpn roles: - role: githubixx.wireguard tags: role-wireguard
vpn is a host group which contains all hosts that WireGuard should be installed on (see Ansible’s
hosts file). E.g.:
[vpn] controller0[1:3].i.domain.tld worker0[1:2].i.domain.tld workstation
Now you can roll out the role:
ansible-playbook --tags=role-wireguard k8s.yml
That limits the tasks which gets executed and in this case that’s the ones from the WireGuard role.
If you use Ansible’s host facts caching as mentioned in part 2 make sure that you refresh the cache afterwards as we now have a new WireGuard interface that Ansible isn’t aware of yet. Use the following command to gather the new host facts:
ansible -m setup all
You should now be able to ping all WireGuard IP’s from your workstation. The same should be true for all hosts of course. You should be able to ping every Kubernetes host from every other Kubernetes node. You just can’t ping your workstation from the Kubernetes nodes as the workstation has no WireGuard endpoint (which was intended).
That’s it for part 3! In part 4 we’ll install a certificate authority which is needed for Kubernetes.