Kubernetes the not so hard way with Ansible (at Scaleway) - Part 3 - Peervpn

December 12, 2016

CHANGELOG

2017-09-10

  • ufw_allow_ports now called harden_linux_ufw_rules in updated harden-linux role.
  • Fix link to Kelsey Hightower’s guide
  • ansible -m setup all should be run after PeerVPN role was applied

This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Provisioning Compute Resources. Since we don’t use AWS or Google Cloud we don’t have the feature set of this two platforms at hand. But we can work around the shortcoming’s with some tools and one of this tool is PeerVPN.

In part 2 we secured our instances. Since we don’t have networking features like AWS VPC or Google Cloud Engine VPC we create our own secure network with PeerVPN. With PeerVPN you can setup a fully meshed (VPN) network even across datacenter. It’s pretty simple to setup and you can even add your workstation to such a network if you like. You only need at least one host which is publicly available on port 7000 (per default, protocol UDP). That’s important e.g. if you want to run hosts at Scaleway without public IPs (which is perfectly possible). So at least you need one host with public IP. Also if you want to include your workstation which is behind a NAT gateway you need one host with a publicly available IP. PeerVPN will do the rest for you.

Setup PeerVPN

I’ve prepared a Ansible role for installing PeerVPN. You can again use

ansible-galaxy install githubixx.peervpn

to install the role or just clone it via git if you like. Basically the PeerVPN settings for all nodes are the same (to be more precise they MUST be the same) - with one exception: peervpn_conf_ifconfig4. Of course every host will get it’s own IP and that’s the variable we use for this setting. For the PeerVPN IP range I choose 10.3.0.0/24. So for the PeerVPN settings that all hosts share we add a few more variable settings to group_vars/k8s.yml with the following content (the values are the default settings):

peervpn_conf_networkname: "peervpn"
peervpn_conf_psk: "preshared_key_change_it"
peervpn_conf_initpeers: "host.example.net 7000"
peervpn_conf_enabletunneling: "yes"
peervpn_conf_interface: "tap0"
peervpn_conf_port: 7000
peervpn_conf_enableipv6: "no"

To create a strong password for the pre-shared key you can create one with

openssl rand -base64 382 | tr -d '\n' && echo

Choose whatever networkname you like for peervpn_conf_networkname parameter. For peervpn_conf_initpeers choose a host with a public IP address and check that port 7000 protocol UDP isn’t blocked by your firewall. All hosts will connect to this host for the initial connection. If you secured your instance as described in part 2 of my tutorial you have to add port 7000 (protocol UDP) to harden_linux_ufw_rules e.g.:

harden_linux_ufw_rules:
  - rule: "allow"
    to_port: "7000"
    protocol: "udp"

For more information about other possible settings have a look at the README of that role but normally you should be fine if you only change the settings I mentioned above.

Finally we need to create a file for every host that is a member of our network. So we create e.g.

host_vars/k8s-controller1.your-domain.tld
host_vars/k8s-controller2.your-domain.tld
host_vars/k8s-controller3.your-domain.tld
host_vars/k8s-worker1.your-domain.tld
host_vars/k8s-worker2.your-domain.tld

and add peervpn_conf_ifconfig4: "10.3.0.xxx/24". Replace xxx with the number/octet you want to use for that host’s PeerVPN IP address and of course use the real hostname for your hosts for the hosts_vars file instead of k8s-XXX.your-domain.tld in my example above. E.g. I usually start at .101 for controller1 and later with .111 for worker1.

Include the role into your playbook like in this example:

- 
  hosts: k8s:children
  roles:
    - 
      role: githubixx.peervpn
      tags: role-peervpn

k8s:children is a group which contains other host groups (in our case that’s the group k8s_controller and k8s_worker). Now you can roll out your PeerVPN role via

ansible-playbook --tags=role-peervpn k8s.yml

That limits the tasks which get’s executed and in this case that’s the ones from the PeerVPN role. It takes a while until the hosts will connect via PeerVPN (check journalctl -f output on the hosts). If you want to apply the role only to one host you can use: ansible-playbook --tags=role-peervpn --limit=k8s-controller1.your-domain.tld k8s.yml (again of course replace k8s-controller1.your-domain.tld with the hostname of your first controller).

If you use Ansibles host facts caching as mentioned in part2 make sure that you refresh the cache as we now have a new PeerVPN interface. Use the following command to gather the new host facts:

ansible -m setup all

That’s it for part 3! In part 4 we’ll install a certificate authority which is needed for Kubernetes.