Kubernetes the not so hard way with Ansible - WireGuard

Install secure, private network for Kubernetes hosts with WireGuard

September 3, 2018

This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Provisioning Compute Resources. Since we don’t use AWS or Google Cloud we don’t have the feature set of this two platforms at hand. But we can work around the shortcoming’s with some tools and one of this tool is WireGuard.

I used PeerVPN before but that wasn’t updated for a while. As I moved my cloud hosts from Scaleway to Hetzner cloud it was a good time to switch the VPN solution ;-) In general PeerVPN still works perfectly fine esp. if you need a easy to setup fully meshed network (where every node is able to talk to all other nodes and even if node A should be able to talk to Node C via node B ;-) ). But PeerVPN needs also lot of CPU resources and throuhput isn’t that good. That’s solved with WireGuard.

In harden the instances we secured our instances. Since we don’t have networking features like AWS VPC or Google Cloud Engine VPC we create our own secure network with WireGuard. All Kubernetes services will only listen on the WireGuard interface (wg0 by default) for incoming requests and all communication between the Kubernetes hosts will be secured by WireGuard VPN. We’ll configure the Ansible WireGuard role that it will create a fully meshed VPN which include at least all Kubernetes hosts. But you can also include additional hosts and even your laptop on which you execute Ansible commands or later even more important the Kubernetes control utility kubectl.

Setup WireGuard

I’ve prepared a Ansible role for installing WireGuard. You can again use

ansible-galaxy install githubixx.wireguard

to install the role or just clone it via git if you like.

By default port 51820 (protocol UDP) should be accessable from the outside. But you can adjust the port by changing the variable wireguard_port in group_vars/all.yml. Also IP forwarding needs to be enabled e.g. via echo 1 > /proc/sys/net/ipv4/ip_forward. I decided not to implement this task in this Ansible role. IMHO that should be handled elsewhere and if you followed my blog series so far my Ansible role harden-linux will handle this for you. See my previous blog post Kubernetes the not so hard way with Ansible - Harden the instances how to implement it. Besides changing sysctl entries (which you need to enable IP forwarding) it also manages firewall settings among other things. As a reminder the following settings should be set in group_vars/all.yml (and don’t forget to roll out the changes before you roll out WireGuard ;-) ):

harden_linux_ufw_rules:
  - rule: "allow"
    to_port: "22222"
    protocol: "tcp"
  - rule: "allow"
    to_port: "51820"
    protocol: "udp"

harden_linux_sysctl_settings_user:
  "net.ipv4.ip_forward": 1
  "net.ipv6.conf.default.forwarding": 1
  "net.ipv6.conf.all.forwarding": 1

harden_linux_ufw_defaults_user:
  "^DEFAULT_FORWARD_POLICY": 'DEFAULT_FORWARD_POLICY="ACCEPT"'

For more information about other possible settings of the Ansible WireGuard role have a look at the README of that role.

You should already have a Ansible host_vars file for every host that is a member of your Kubernetes cluster and maybe also for your workstation you execute ansible-playbook and kubectl later e.g.:

host_vars/controller01.i.domain.tld
host_vars/controller02.i.domain.tld
host_vars/controller03.i.domain.tld
host_vars/worker01.i.domain.tld
host_vars/worker02.i.domain.tld
host_vars/workstation

In Kubernetes the not so hard way with Ansible - Harden the instances I already showed you examples what settings you should set for every host so I won’t repeat everything here again. Just the example for the first controller node:

Ansible host file: host_vars/controller01.i.domain.tld:

---
wireguard_address: "10.8.0.101/24"
wireguard_endpoint: "controller01.p.domain.tld"
ansible_host: "controller01.p.domain.tld"
ansible_python_interpreter: /usr/bin/python3

You need to set wireguard_address for every node. Of course every node needs a different IP but make sure to use the same network like /24 in the example above. You also need to set wireguard_endpoint for every node that will be part of your Kubernetes cluster. Only the workstation will have a wireguard_endpoint set to "" (empty string). The reason for this is that the workstation won’t have a WireGuard VPN endpoint where the other hosts would connect to. You just want to access the Kubernetes cluster from your workstation and not the other way round. But you want to enable all Kubernetes hosts to communicate with each other. That’s what wireguard_endpoint is needed. Besides other things the Ansible role will create a WireGuard configuration for every host. That configuration will include the information what WireGuard IP can be reached via what endpoint. You can find a complete example in the README of the WireGuard role but here is a example of the WireGurad configuration of the workstation:

[Interface]
Address = 10.8.0.2
PrivateKey = ....
ListenPort = 51820

[Peer]
PrivateKey = ....
AllowedIPs = 10.8.0.101/24
Endpoint = controller01.p.domain.tld:51820

[Peer]
PrivateKey = ....
AllowedIPs = 10.8.0.102/24
Endpoint = controller02.p.domain.tld:51820

...

The AllowedIPs parameter is also some kind of routing information and WireGuard will create routes accordingly. So on my workstation with the IP 10.8.0.2 there will be a route that tells WireGuard that if I want to connect to 10.8.0.101 (which is the first Kubernetes controller node) send the traffic to controller01.p.domain.tld:51820.

Include the role into your playbook like in this example:

-
  hosts: vpn
  roles:
    -
      role: githubixx.wireguard
      tags: role-wireguard

vpn is a host group which contains all hosts that WireGuard should be installed on (see Ansible’s hosts file). E.g.:

[vpn]
controller0[1:3].i.domain.tld
worker0[1:2].i.domain.tld
workstation

Now you can roll out the role:

ansible-playbook --tags=role-wireguard k8s.yml

That limits the tasks which get’s executed and in this case that’s the ones from the WireGuard role. If you want to apply the role only to one host you can use:

ansible-playbook --tags=role-wireguard --limit=controller01.i.domain.tld k8s.yml

(again of course replace controller01.i.domain.tld with a real hostname).

If you use Ansibles host facts caching as mentioned in part 2 make sure that you refresh the cache afterwards as we now have a new WireGuard interface. Use the following command to gather the new host facts:

ansible -m setup all

You should now be able to ping all WireGuard IP’s from your workstation. The same should be true for all hosts of course. You should be able to ping every Kubernetes host from every Kubernetes node. You just can’t ping your workstation from the Kubernetes nodes as the workstation has no WireGuard endpoint.

That’s it for part 3! In part 4 we’ll install a certificate authority which is needed for Kubernetes.