Kubernetes the not so hard way with Ansible (at Scaleway) - Part 6 - Control plane [updated for K8s v1.8]

February 15, 2017

CHANGELOG

2017-11-22

2017-10-09

  • k8s_binaries renamed to k8s_controller_binaries
  • make systemd service template files for API server, controller manager and schedule more flexible by allowing more parameter to be changed via variable
  • k8s_auth_tokens no longer used as we now use Role-Based Access Control (“RBAC”) everywhere
  • RBAC for kubelet authorization

This post is based on Kelsey Hightower’s Bootstrapping the Kubernetes Control Plane.

This time we install a 3 node Kubernetes controller cluster (that’s Kubernetes API Server, Scheduler and Controller manager). All components will run on every node for HA. In part 4 we installed our PKI (public key infrastructure) in order to secure communication between our Kubernetes components/infrastructure. As with the etcd-cluster we use the certificate authority and generated certificates but for Kubernetes API server we generated a separate CA and certificate. If you used the default values in the other playbooks so far you most likely don’t need to change any default variable setting which are:

k8s_conf_dir: "/var/lib/kubernetes"
k8s_bin_dir: "/usr/local/bin"
k8s_release: "1.8.0"
k8s_interface: "tap0"

k8s_ca_conf_directory: "/etc/k8s/certs"
k8s_config_directory: "/etc/k8s/configs"

k8s_controller_binaries:
  - kube-apiserver
  - kube-controller-manager
  - kube-scheduler
  - kubectl

k8s_certificates:
  - ca-k8s-apiserver.pem
  - ca-k8s-apiserver-key.pem
  - cert-k8s-apiserver.pem
  - cert-k8s-apiserver-key.pem

k8s_apiserver_admission_control: "Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
k8s_apiserver_allow_privileged: "true"
k8s_apiserver_apiserver_count: "3"
k8s_apiserver_authorization_mode: "Node,RBAC"
k8s_apiserver_audit_log_maxage: "30"
k8s_apiserver_audit_log_maxbackup: "3"
k8s_apiserver_audit_log_maxsize: "100"
k8s_apiserver_audit_log_path: "/var/log/audit.log"
k8s_apiserver_enable_swagger_ui: "true"
k8s_apiserver_event_ttl: "1h"
k8s_apiserver_kubelet_https: "true"
k8s_apiserver_runtime_config: "api/all"
k8s_apiserver_kubelet_preferred_address_types: "InternalIP,Hostname,ExternalIP"
k8s_apiserver_service_cluster_ip_range: "10.32.0.0/16"
k8s_apiserver_service_node_port_range: "30000-32767"

k8s_controller_manager_cluster_cidr: "10.200.0.0/16"
k8s_controller_manager_cluser_name: "kubernetes"
k8s_controller_manager_leader_elect: "true"
k8s_controller_manager_conf_dir: "{{k8s_conf_dir}}"
k8s_controller_manager_service_cluster_ip_range: "{{k8s_apiserver_service_cluster_ip_range}}"

k8s_scheduler_leader_elect: "{{k8s_controller_manager_leader_elect}}"

etcd_client_port: "2379"
etcd_interface: "tap0"

etcd_certificates:
  - ca-etcd.pem
  - ca-etcd-key.pem
  - cert-etcd.pem
  - cert-etcd-key.pem

As you can see we install Kubernetes 1.8.0 by default. The role will search for the certificates we created in part 4 in the directory you specify in k8s_ca_conf_directory on the host you run Ansible. Also the encryption file will be used which this role should find in k8s_config_directory. The CA and certificate files used here are listed in k8s_certificates. The binaries listed in k8s_controller_binaries will be downloaded and stored into the directory you specify in k8s_bin_dir. If you created a different interface for PeerVPN (e.g. peervpn0) change k8s_interface.

If you ask yourself “why do we need to specify etcd_certificates here again?”: Well the Kubernetes API server needs to communicate with the Kubernetes componentes AND the etcd cluster as you may remember. That’s the reason why it must be aware of both CA’s and certificates. But since we store all group variables in group_vars/k8s.yml it’s of course sufficient to specifiy all variables only once there even if you see the same variable in different roles (mainly in defaults/main.yml).

Now add an entry for your controller hosts into Ansible’s hosts file e.g. (of course you need to change controller[1:3].your.tld to your own hostnames:

[k8s_controller]
controller[1:3].your.tld

Install the role via

ansible-galaxy install githubixx.kubernetes-controller

Next add the role ansible-role-kubernetes-controller to the k8s.yml file e.g.:

  hosts: k8s_controller
  roles:
    -
      role: githubixx.kubernetes-controller 
      tags: role-kubernetes-controller

Apply the role via

ansible-playbook --tags=role-kubernetes-controller k8s.yml

After the role is applyed you can check the status of the components with kubectl get componentstatuses but first we need to configure kubectl. We alredy installed kubectl locally in part 2 of my tutorial. I’ve prepared a playbook to do the kubectl configuration. You should already have cloned my ansible-kubernetes-playbooks repository. I recommend to place it at the same directory level as Ansible’s roles directory (git clone https://github.com/githubixx/ansible-kubernetes-playbooks). Switch to ansible-kubernetes-playbooks/kubectlconfig directory. There is now one thing you may need to change: https://github.com/githubixx/ansible-kubernetes-playbooks/blob/6704c2368526b6a53a04a599eb22299b39d28116/kubectlconfig/kubectlconfig.yml#L11 . This complicated looking line get’s the first hostname in our [k8s_controller] host group and uses the IP address of this host’s PeerVPN interface as the API server address for kubectl (kubectl is basically the frontend utility for the API server). My laptop also has PeerVPN installed and it’s part of this Kubernetes PeerVPN mesh network. This allow’s kubectl on my laptop to contact the API server. But that may not work for you. Either do the same or you maybe setup ssh forwarding to one of the controller node’s PeerVPN interface (port 6443 by default) and then use --server=https://localhost:6443 or you do something completly different ;-) You could also copy $HOME/.kube directory (if the configs are generated in a moment) to one of the Kubernetes hosts and work from there.

Now generate the kubectl configuration with

ansible-playbook kubectlconfig.yml

If you have your Ansible variables all in place as I suggested in my previous posts it should just work. The playbook will configure kubectl using the admin certificates we created with the Ansible role role-kubernetes-ca.

If you now run kubectl get componentstatuses one would expect to see this output:

kubectl get componentstatuses

NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}  

BUT instead you will probably see this:

NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused   
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused   
etcd-0               Healthy     {"health": "true"}                                                                             
etcd-1               Healthy     {"health": "true"}                                                                             
etcd-2               Healthy     {"health": "true"}

If you don’t see any error in systemd’s journal ( journalctl --no-pager ) on the controller nodes and the scheduler and controller-manager are running and listening ( netstat -tlpn ) on port 10251 and 10252 and you get the output above it’s because of this long standig bug “kubectl get cs”: incorrect hard coded master component locations. ATM this can only be avoided if you bind the scheduler and the controller-manager to 0.0.0.0 which means to listen on all interfaces. But this is something I don’t want. I configured this services to listen on the PeerVPN interface because it’s really sufficient. If you don’t care then change this line in the controller manager systemd service template and this line in the scheduler systemd service template to 0.0.0.0 (even 127.0.0.1 won’t work). An alternative would be to setup a iptables rule to forward the traffic accordingly (haven’t looked at this yet).

Now it’s time to setup the Kubernetes worker.