TauCeti.blog
Posts Tags
TauCeti.blog

Kubernetes the not so hard way with Ansible - Wireguard - (K8s v1.27)

     1189 words   6 minutes 

2023-09-10

  • fixed outdated URLS

2020-05-08

2020-01-26

  • added comment about setting IP forwarding via wireguard_preup

This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Provisioning Compute Resources. Since I don’t use AWS or Google Cloud I don’t have the feature set of this two platforms at hand. But we can work around the shortcoming’s with some tools and one of this tool is Wireguard.

One hint about Hetzner Cloud Networks: Just because they are private doesn’t mean that they are secure 😉 Traffic is NOT encrypted in that case. So for secure communication you still need something like Wireguard. And if you want connect your laptop at home with your servers somewhere a VPN is also quite handy ;-)

I used PeerVPN before but that wasn’t updated for a while. As I moved my cloud hosts from Scaleway to Hetzner cloud it was a good time to switch the VPN solution ;-) In general PeerVPN still works perfectly fine esp. if you need a easy to setup fully meshed network (where every node is able to talk to all other nodes and even if node A should be able to talk to Node C via node B ;-) ). But PeerVPN needs also lot of CPU resources and throughput isn’t that good. That’s solved with Wireguard.

In harden the instances I secured the VM instances. Since we don’t have networking features like AWS VPC or Google Cloud Engine VPC I create a secure network with Wireguard. All Kubernetes services will only listen on the Wireguard interface (wg0 by default) for incoming requests and all communication between the Kubernetes hosts will be secured by Wireguard VPN. We’ll configure the Ansible Wireguard role to create a fully meshed VPN which includes at least all Kubernetes hosts. But you can also include additional hosts and even your laptop on which you execute Ansible commands or later even more important the Kubernetes control utility kubectl.

I’ve prepared a Ansible role for installing Wireguard. You can again use

ansible-galaxy install githubixx.ansible_role_wireguard

to install the role or just clone it via git if you like.

By default port 51820 (protocol UDP) should be accessible from the outside. But you can adjust the port by changing the variable wireguard_port in group_vars/all.yml or in host_vars for a specific host. Also IP forwarding needs to be enabled e.g. via echo 1 > /proc/sys/net/ipv4/ip_forward. I decided not to implement this tasks in this Ansible role. IMHO that should be handled elsewhere. But you can use the wireguard_(preup|predown|postup|postdown) hooks for such kind of tasks. E.g. you can define in hosts_vars (or group_vars/all.yml) for one, some or all hosts something like this:

wireguard_preup:
  - echo 1 > /proc/sys/net/ipv4/ip_forward
  - ufw allow 51820/udp

This would enable IP forwarding and open port 51820 for UDP protocol before the Wireguard interfaces are created (in this case I assumed that UFW - Uncomplicated Firewall is used as an iptables fronted which is the case if you use harden_linux).

If you followed my blog series so far then you already know that my Ansible role harden_linux is also able to do this for you. See my previous blog post Kubernetes the not so hard way with Ansible - Harden the instances how to implement it. Besides changing sysctl entries (which you need to enable IP forwarding) it also manages firewall settings among other things. As a reminder the following settings should be set in group_vars/all.yml (and don’t forget to roll out the changes before you roll out Wireguard ;-) ):

harden_linux_ufw_rules:
  - rule: "allow"
    to_port: "22222"
    protocol: "tcp"
  - rule: "allow"
    to_port: "51820"
    protocol: "udp"

harden_linux_sysctl_settings_user:
  "net.ipv4.ip_forward": 1
  "net.ipv6.conf.default.forwarding": 1
  "net.ipv6.conf.all.forwarding": 1

harden_linux_ufw_defaults_user:
  "^DEFAULT_FORWARD_POLICY": 'DEFAULT_FORWARD_POLICY="ACCEPT"'

For more information about other possible settings of the Ansible Wireguard role have a look at the README of that role.

You should already have a Ansible host_vars file for every host that is a member of your Kubernetes cluster and maybe also for your workstation you execute ansible-playbook and kubectl later e.g.:

host_vars/controller01.i.domain.tld
host_vars/controller02.i.domain.tld
host_vars/controller03.i.domain.tld
host_vars/worker01.i.domain.tld
host_vars/worker02.i.domain.tld
host_vars/workstation

In Kubernetes the not so hard way with Ansible - Harden the instances I already showed you examples what settings you should set for every host so I won’t repeat everything here again. Just the example for the first controller node:

Ansible host file: host_vars/controller01.i.domain.tld:

---
wireguard_address: "10.8.0.101/24"
wireguard_port: 51820
wireguard_endpoint: "controller01.p.domain.tld"
wireguard_persistent_keepalive: "30"

ansible_host: "controller01.p.domain.tld"
ansible_port: 22222
ansible_user: dauser
ansible_become: true
ansible_become_method: sudo
ansible_python_interpreter: /usr/bin/python3

You need to set wireguard_address for every node. Of course every node needs a different IP but make sure to use the same network like /24 in the example above. You also need to set wireguard_endpoint for every node that will be part of your Kubernetes cluster. Only the workstation will have a wireguard_endpoint set to "" (empty string). The reason for this is that the workstation won’t have a Wireguard VPN endpoint where the other hosts would connect to. You just want to access the Kubernetes cluster from your workstation and not the other way round. But you want to enable all Kubernetes hosts to communicate with each other. That’s what wireguard_endpoint is needed for. Besides other things the Ansible role will create a Wireguard configuration for every host. That configuration will include the information what Wireguard IP can be reached via what endpoint. You can find a complete example in the README of the Wireguard role but here is a example of the Wireguard configuration of the workstation:

[Interface]
Address = 10.8.0.2/24
PrivateKey = ....
ListenPort = 51820

[Peer]
PrivateKey = ....
AllowedIPs = 10.8.0.101/32
Endpoint = controller01.p.domain.tld:51820

[Peer]
PrivateKey = ....
AllowedIPs = 10.8.0.102/32
Endpoint = controller02.p.domain.tld:51820
...

The AllowedIPs parameter is also some kind of routing information and Wireguard will create routes accordingly. So on my workstation with the IP 10.8.0.2 there will be a route that tells Wireguard that if I want to connect to 10.8.0.101 (which is the first Kubernetes controller node) send the traffic to controller01.p.domain.tld:51820.

Include the role into your playbook like in this example:

-
  hosts: vpn
  roles:
    -
      role: githubixx.wireguard
      tags: role-wireguard

vpn is a host group which contains all hosts that Wireguard should be installed on (see Ansible’s hosts file). E.g.:

[vpn]
controller0[1:3].i.domain.tld
worker0[1:2].i.domain.tld
workstation

Now you can roll out the role:

ansible-playbook --tags=role-wireguard k8s.yml

That limits the tasks which gets executed and in this case that’s the ones from the Wireguard role.

If you use Ansible’s host facts caching as mentioned in part 2 make sure that you refresh the cache afterwards as we now have a new Wireguard interface that Ansible isn’t aware of yet. Use the following command to gather the new host facts:

ansible -m setup all

You should now be able to ping all Wireguard IP’s from your workstation. The same should be true for all hosts of course. You should be able to ping every Kubernetes host from every other Kubernetes node. You just can’t ping your workstation from the Kubernetes nodes as the workstation has no Wireguard endpoint (which was intended).

That’s it for part 3! In part 4 we’ll install a certificate authority which is needed for Kubernetes.

Updated on 2023-09-10
Back | Home