Kubernetes the not so hard way with Ansible - Control plane (updated for K8s v1.11.x)

Install and configure the heart of Kubernetes: kube-apiserver, kube-controller-manager and kube-scheduler

September 6, 2018



  • upgrade to k8s_release to 1.11.3 for Kubernetes v1.11.3

This post is based on Kelsey Hightower’s Bootstrapping the Kubernetes Control Plane.

This time we install a 3 node Kubernetes controller cluster (that’s Kubernetes API Server, Scheduler and Controller manager). All this components will run on every node. In Kubernetes certificate authority we installed our PKI (public key infrastructure) in order to secure communication between our Kubernetes components/infrastructure. As with the etcd-cluster we use the certificate authority and generated certificates but for Kubernetes API server we generated a separate CA and certificate. If you used the default values in the other playbooks so far you most likely don’t need to change any default variable setting which are:

# The directory to store the K8s certificates and other configuration
k8s_conf_dir: "/var/lib/kubernetes"
# The directory to store the K8s binaries
k8s_bin_dir: "/usr/local/bin"
# K8s release
k8s_release: "1.11.3"
# The interface on which the K8s services should listen on. As all cluster
# communication should use a VPN interface the interface name is
# normally "wg0" (WireGuard),"peervpn0" (PeerVPN) or "tap0".
k8s_interface: "wg0"

# The directory from where to copy the K8s certificates. By default this
# will expand to user's LOCAL $HOME (the user that run's "ansible-playbook ..."
# plus "/k8s/certs". That means if the user's $HOME directory is e.g.
# "/home/da_user" then "k8s_ca_conf_directory" will have a value of
# "/home/da_user/k8s/certs".
k8s_ca_conf_directory: "{{ '~/k8s/certs' | expanduser }}"
# Directory where kubeconfig for Kubernetes worker nodes and kube-proxy
# is stored among other configuration files. Same variable expansion
# rule applies as with "k8s_ca_conf_directory"
k8s_config_directory: "{{ '~/k8s/configs' | expanduser }}"

# K8s control plane binaries to download
  - kube-apiserver
  - kube-controller-manager
  - kube-scheduler
  - kubectl

# K8s kube-(apiserver|controller-manager-sa) certificates
  - ca-k8s-apiserver.pem
  - ca-k8s-apiserver-key.pem
  - cert-k8s-apiserver.pem
  - cert-k8s-apiserver-key.pem
  - cert-k8s-controller-manager-sa.pem
  - cert-k8s-controller-manager-sa-key.pem

k8s_apiserver_secure_port: "6443"

# K8s API daemon settings (can be overriden or additional added by defining
# "k8s_apiserver_settings_user")
  "advertise-address": "{{hostvars[inventory_hostname]['ansible_' + k8s_interface].ipv4.address}}"
  "bind-address": "{{hostvars[inventory_hostname]['ansible_' + k8s_interface].ipv4.address}}"
  "secure-port": "{{k8s_apiserver_secure_port}}"
  "enable-admission-plugins": "Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
  "allow-privileged": "true"
  "apiserver-count": "3"
  "authorization-mode": "Node,RBAC"
  "audit-log-maxage": "30"
  "audit-log-maxbackup": "3"
  "audit-log-maxsize": "100"
  "audit-log-path": "/var/log/audit.log"
  "enable-swagger-ui": "true"
  "event-ttl": "1h"
  "kubelet-https": "true"
  "kubelet-preferred-address-types": "InternalIP,Hostname,ExternalIP" # "--kubelet-preferred-address-types" defaults to:
                                                                      # "Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP"
                                                                      # Needs to be changed to make "kubectl logs" and "kubectl exec" work.
  "runtime-config": "api/all"
  "service-cluster-ip-range": ""
  "service-node-port-range": "30000-32767"
  "client-ca-file": "{{k8s_conf_dir}}/ca-k8s-apiserver.pem"
  "etcd-cafile": "{{k8s_conf_dir}}/ca-etcd.pem"
  "etcd-certfile": "{{k8s_conf_dir}}/cert-etcd.pem"
  "etcd-keyfile": "{{k8s_conf_dir}}/cert-etcd-key.pem"
  "experimental-encryption-provider-config": "{{k8s_conf_dir}}/encryption-config.yaml"
  "kubelet-certificate-authority": "{{k8s_conf_dir}}/ca-k8s-apiserver.pem"
  "kubelet-client-certificate": "{{k8s_conf_dir}}/cert-k8s-apiserver.pem"
  "kubelet-client-key": "{{k8s_conf_dir}}/cert-k8s-apiserver-key.pem"
  "service-account-key-file": "{{k8s_conf_dir}}/cert-k8s-controller-manager-sa.pem"
  "tls-cert-file": "{{k8s_conf_dir}}/cert-k8s-apiserver.pem"
  "tls-private-key-file": "{{k8s_conf_dir}}/cert-k8s-apiserver-key.pem"

# The directory to store controller manager configuration.
k8s_controller_manager_conf_dir: "/var/lib/kube-controller-manager"
# K8s controller manager settings (can be overriden or additional added by defining
# "k8s_controller_manager_settings_user")
  "address": "{{hostvars[inventory_hostname]['ansible_' + k8s_interface].ipv4.address}}"
  "cluster-cidr": ""
  "cluster-name": "kubernetes"
  "kubeconfig": "{{k8s_controller_manager_conf_dir}}/kube-controller-manager.kubeconfig"
  "leader-elect": "true"
  "service-cluster-ip-range": ""
  "cluster-signing-cert-file": "{{k8s_conf_dir}}/ca-k8s-apiserver.pem"
  "cluster-signing-key-file": "{{k8s_conf_dir}}/cert-k8s-apiserver-key.pem"
  "root-ca-file": "{{k8s_conf_dir}}/ca-k8s-apiserver.pem"
  "service-account-private-key-file": "{{k8s_conf_dir}}/cert-k8s-controller-manager-sa-key.pem"
  "use-service-account-credentials": "true"

# The directory to store scheduler configuration.
k8s_scheduler_conf_dir: "/var/lib/kube-scheduler"
# kube-scheduler settings (only --config left,
# see https://github.com/kubernetes/kubernetes/pull/62515)
  "config": "{{k8s_scheduler_conf_dir}}/kube-scheduler.yaml"

# The port the control plane componentes should connect to etcd cluster
etcd_client_port: "2379"
# The interface the etcd cluster is listening on
etcd_interface: "wg0"

# The etcd certificates needed for the control plane componentes to be able
# to connect to the etcd cluster.
  - ca-etcd.pem
  - ca-etcd-key.pem
  - cert-etcd.pem
  - cert-etcd-key.pem

The kube-apiserver settings defined in k8s_apiserver_settings can be overriden by defining a variable called k8s_apiserver_settings_user. You can also add additional settings for the kube-apiserver daemon by using this variable. E.g. to override audit-log-maxage and audit-log-maxbackup default values and add watch-cache option add the following settings to group_vars/all.yml (or where ever it fit’s best for you):

  "audit-log-maxage": "40"
  "audit-log-maxbackup": "4"
  "watch-cache": "false"

The same is true for the kube-controller-manager by adding entries to k8s_controller_manager_settings_user variable. For kube-scheduler add entries to k8s_scheduler_settings_user variable to override settings in k8s_scheduler_settings dictionary or to add new one.

As you can see we install Kubernetes 1.11.x by default. The role will search for the certificates we created in certificate authority in the directory you specified in k8s_ca_conf_directory on the host you run Ansible. Also the encryption file will be used which this role should find in k8s_encryption_config_directory (which is the same as k8s_config_directory in my case). The CA and certificate files used here are listed in k8s_certificates. The binaries listed in k8s_controller_binaries will be downloaded and stored into the directory you specify in k8s_bin_dir. If you followed my guide so far the interface for the VPN is again wg0 for k8s_interface.

If you ask yourself “why do we need to specify etcd_certificates here again?”: Well the Kubernetes API server needs to communicate with the Kubernetes componentes AND the etcd cluster as you may remember. That’s the reason why it must be aware of both CA’s and certificates. But since we store all group variables in group_vars/all.yml it’s of course sufficient to specifiy all variables only once there even if you see the same variable in different roles. Just get sure you’ve set the variables to a valid value.

Now add an entry for your controller hosts into Ansible’s hosts file e.g. (of course you need to change controller0[1:3].i.domain.tld to your own hostnames):


Install the role via

ansible-galaxy install githubixx.kubernetes-controller

Next add the role ansible-role-kubernetes-controller to the k8s.yml playbook file e.g.:

  hosts: k8s_controller
      role: githubixx.kubernetes-controller 
      tags: role-kubernetes-controller

Apply the role via

ansible-playbook --tags=role-kubernetes-controller k8s.yml

After the role is applyed you can basically check the status of the components with:

kubectl get componentstatuses

BUT first we need to configure kubectl ;-) We already installed kubectl locally in harden the instances of my tutorial. I’ve prepared a playbook to do the kubectl configuration. You should already have cloned my ansible-kubernetes-playbooks repository. I recommend to place it at the same directory level as Ansible’s roles directory (git clone https://github.com/githubixx/ansible-kubernetes-playbooks). Switch to ansible-kubernetes-playbooks/kubectlconfig directory.

There is now one thing you may need to change: https://github.com/githubixx/ansible-kubernetes-playbooks/blob/master/kubectlconfig/kubectlconfig.yml#L11 . This complicated looking line get’s the first hostname in our [k8s_controller] host group and uses the IP address of this host’s VPN interface as the API server address for kubectl (kubectl is basically the frontend utility for the API server). My laptop has WireGuard installed and it’s part of this Kubernetes WireGuard VPN fully meshed network. This allow’s kubectl on my laptop to contact the API server.

But that may not work for you if your workstation is not part the WireGuard VPN. Either do the same or you maybe setup ssh forwarding to one of the controller node’s VPN interface (port 6443 by default) and then use --server=https://localhost:6443 or you do something completly different ;-) You could also copy $HOME/.kube directory (if the configs are generated in a moment) to one of the Kubernetes hosts and work from there.

Now generate the kubectl configuration with

ansible-playbook kubectlconfig.yml

If you have your Ansible variables all in place as I suggested in my previous posts it should just work. The playbook will configure kubectl using the admin certificates we created with the Ansible role role-kubernetes-ca.

If you now run kubectl get componentstatuses one would expect to see this output:

kubectl get componentstatuses

NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}  

BUT instead you will probably see this:

NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get dial tcp getsockopt: connection refused   
controller-manager   Unhealthy   Get dial tcp getsockopt: connection refused   
etcd-0               Healthy     {"health": "true"}                                                                             
etcd-1               Healthy     {"health": "true"}                                                                             
etcd-2               Healthy     {"health": "true"}

If you don’t see any error in systemd’s journal ( journalctl --no-pager ) on the controller nodes and the scheduler and controller-manager are running and listening ( netstat -tlpn ) on port 10251 and 10252 and you get the output above it’s because of this long standig bug “kubectl get cs”: incorrect hard coded master component locations. ATM this can only be avoided if you bind the scheduler and the controller-manager to which means to listen on all interfaces. But this is something I don’t want. I configured this services to listen only on the VPN interface (wg0 e.g.) because it’s really sufficient. This way all communication is secure automatically because the encrypted VPN connection and more important the VPN interfaces shouldn’t be reachable from the Internet by default even if you don’t use any firewall. And just because of this bug I should make scheduler and controller-manager listening on all interfaces? Doesn’t really makes sense for me…

If you don’t care about the issues mentioned above you can use different defaults for scheduler and controller-manager by defining

  "address": ""


  "address": ""

in group_vars/all.yml.

An alternative would be to setup a iptables rule to forward the traffic accordingly (haven’t looked at this yet).

Now it’s time to setup the Kubernetes worker.