Kubernetes the not so hard way with Ansible - etcd cluster - (K8s v1.28)
This post is based on Kelsey Hightower’s Kubernetes The Hard Way - Bootstrapping the etcd cluster.
In the previous part certificate authority I installed the PKI (public key infrastructure) in order to secure communication between our Kubernetes components/infrastructure and also created certificates for authentication of the various K8s components. Now we use the certificate authorities (CA) and generated keys for the first and very important component - the etcd cluster. etcd is basically a distributed key/value database. The Kubernetes services in general are stateless. All state is stored in etcd so you should take care of your etcd cluster in production. If you loose all etcd nodes you loose the whole Kubernetes state… So making a snapshot/backup from time to time is - at least - recommended 😉
I want to mention that if the etcd cluster nodes wont join then a possible reason could be the certificates. If it isn’t the firewall blocking traffic between etcd nodes, the certificate’s host list could be the problem. The error message isn’t always clear about the issue.
As usual we add the role ansible-role-etcd to the k8s.yml file e.g.:
hosts: k8s_etcd
roles:
-
role: githubixx.etcd
tags: role-etcd
Next install the role via
ansible-galaxy install githubixx.etcd
(or just clone the Github repo whatever you like). Basically you don’t need to change a lot of variables but you can if you want of course:
# The directory from where to copy the etcd certificates. By default this
# will expand to user's LOCAL $HOME (the user that run's "ansible-playbook ..."
# plus "/etcd-certificates". That means if the user's $HOME directory is e.g.
# "/home/da_user" then "etcd_ca_conf_directory" will have a value of
# "/home/da_user/etcd-certificates".
etcd_ca_conf_directory: "{{ '~/etcd-certificates' | expanduser }}"
# etcd Ansible group
etcd_ansible_group: "k8s_etcd"
# etcd version
etcd_version: "3.5.9"
# Port where etcd listening for clients
etcd_client_port: "2379"
# Port where etcd is listening for it's peer's
etcd_peer_port: "2380"
# Interface to bind etcd ports to
etcd_interface: "tap0"
# Run etcd daemon as this user.
#
# Note 1: If you want to use an "etcd_peer_port" < 1024 you most probably need
# to run "etcd" as user "root".
# Note 2: If the user specified in "etcd_user" does not exist then the role
# will create it. Only if the user already exists the role will not create it
# but it will adjust it's UID/GID and shell if specified (see settings below).
# Additionally if "etcd_user" is "root" then this role wont touch the user
# at all.
etcd_user: "etcd"
# UID of user specified in "etcd_user". If not specified the next available
# UID from "/etc/login.defs" will be taken (see "SYS_UID_MAX" setting).
# etcd_user_uid: "999"
# Shell for specified user in "etcd_user". For increased security keep
# the default.
etcd_user_shell: "/bin/false"
# Specifies if the user specified in "etcd_user" will be a system user (default)
# or not. If "true" the "etcd_user_home" setting will be ignored. In general
# it makes sense to keep the default as there should be no need to login as
# the user that runs "etcd".
etcd_user_system: true
# Home directory of user specified in "etcd_user". Will be ignored if
# "etcd_user_system" is set to "true". In this case no home directory will
# be created. Normally not needed.
# etcd_user_home: "/home/etcd"
# Run etcd daemon as this group
#
# Note: If the group specified in "etcd_group" does not exist then the role
# will create it. Only if the group already exists the role will not create it
# but will adjust GID if specified in "etcd_group_gid" (see setting below).
etcd_group: "etcd"
# GID of group specified in "etcd_group". If not specified the next available
# GID from "/etc/login.defs" will be take (see "SYS_GID_MAX" setting).
# etcd_group_gid: "999"
# Specifies if the group specified in "etcd_group" will be a system group (default)
# or not.
etcd_group_system: true
# Directory for etcd configuration
etcd_conf_dir: "/etc/etcd"
# Permissions for directory for etcd configuration
etcd_conf_dir_mode: "0750"
# Owner of directory specified in "etcd_conf_dir"
etcd_conf_dir_user: "root"
# Group owner of directory specified in "etcd_conf_dir"
etcd_conf_dir_group: "{{ etcd_group }}"
# Directory to store downloaded etcd archive
# Should not be deleted to avoid downloading over and over again
etcd_download_dir: "/opt/etcd"
# Permissions for directory to store downloaded etcd archive
etcd_download_dir_mode: "0755"
# Owner of directory specified in "etcd_download_dir"
etcd_download_dir_user: "{{ etcd_user }}"
# Group owner of directory specified in "etcd_download_dir"
etcd_download_dir_group: "{{ etcd_group }}"
# Directory to store etcd binaries
#
# IMPORTANT: If you use the default value for "etcd_bin_dir" which is
# "/usr/local/bin" then the settings specified in "etcd_bin_dir_mode",
# "etcd_bin_dir_user" and "etcd_bin_dir_group" are ignored. This is
# done to prevent that the permissions of "/usr/local/bin" are changed.
# This directory normally exists already on every Linux installation
# and should not be changed.
# So please be careful if you specify a directory like "/usr/bin" or
# "/bin" as "etcd_bin_dir" as this will change the permissions of
# these directories and this is something you normally do not want.
etcd_bin_dir: "/usr/local/bin"
# Permissions for directory to store etcd binaries
etcd_bin_dir_mode: "0755"
# Owner of directory specified in "etcd_bin_dir"
etcd_bin_dir_user: "{{ etcd_user }}"
# Group owner of directory specified in "etcd_bin_dir"
etcd_bin_dir_group: "{{ etcd_group }}"
# etcd data directory (etcd database files so to say)
etcd_data_dir: "/var/lib/etcd"
# Permissions for directory to store etcd data
etcd_data_dir_mode: "0700"
# Owner of directory specified in "etcd_data_dir"
etcd_data_dir_user: "{{ etcd_user }}"
# Group owner of directory specified in "etcd_data_dir"
etcd_data_dir_group: "{{ etcd_group }}"
# Architecture to download and install
etcd_architecture: "amd64"
# Only change this if the architecture you are using is unsupported
# For more information, see this:
# https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/op-guide/supported-platform.md
etcd_allow_unsupported_archs: false
# By default etcd tarball gets downloaded from official
# etcd repository. This can be changed to some custom
# URL if needed. For more information which protocols
# can be used see:
# https://docs.ansible.com/ansible/latest/collections/ansible/builtin/get_url_module.html
# It's only important to keep the filename naming schema:
# "etcd-v{{ etcd_version }}-linux-{{ etcd_architecture }}.tar.gz"
etcd_download_url: "https://github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-{{ etcd_architecture }}.tar.gz"
# By default the SHA256SUMS file is used to verify the
# checksum of the tarball archive. This can also be
# changed to your needs.
etcd_download_url_checksum: "sha256:https://github.com/coreos/etcd/releases/download/v{{ etcd_version }}/SHA256SUMS"
# Options for [Service] section. For more information see:
# https://www.freedesktop.org/software/systemd/man/systemd.service.html#Options
# The options below "Type=notify" are mostly security/sandbox related settings
# and limit the exposure of the system towards the unit's processes.
# https://www.freedesktop.org/software/systemd/man/systemd.exec.html
etcd_service_options:
- User={{ etcd_user }}
- Group={{ etcd_group }}
- Restart=on-failure
- RestartSec=5
- Type=notify
- ProtectHome=true
- PrivateTmp=true
- ProtectSystem=full
- ProtectKernelModules=true
- ProtectKernelTunables=true
- ProtectControlGroups=true
- CapabilityBoundingSet=~CAP_SYS_PTRACE
etcd_settings:
"name": "{{ ansible_hostname }}"
"cert-file": "{{ etcd_conf_dir }}/cert-etcd-server.pem"
"key-file": "{{ etcd_conf_dir }}/cert-etcd-server-key.pem"
"trusted-ca-file": "{{ etcd_conf_dir }}/ca-etcd.pem"
"peer-cert-file": "{{ etcd_conf_dir }}/cert-etcd-peer.pem"
"peer-key-file": "{{ etcd_conf_dir }}/cert-etcd-peer-key.pem"
"peer-trusted-ca-file": "{{ etcd_conf_dir }}/ca-etcd.pem"
"advertise-client-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_client_port }}"
"initial-advertise-peer-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_peer_port }}"
"listen-peer-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_peer_port }}"
"listen-client-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_client_port + ',https://127.0.0.1:' + etcd_client_port }}"
"peer-client-cert-auth": "true" # Enable peer client cert authentication
"client-cert-auth": "true" # Enable client cert authentication
"initial-cluster-token": "etcd-cluster-0" # Initial cluster token for the etcd cluster during bootstrap.
"initial-cluster-state": "new" # Initial cluster state ('new' or 'existing')
"data-dir": "{{ etcd_data_dir }}" # etcd data directory (etcd database files so to say)
"wal-dir": "" # Dedicated wal directory ("" means no separated WAL directory)
"auto-compaction-retention": "0" # Auto compaction retention in hour. 0 means disable auto compaction.
"snapshot-count": "100000" # Number of committed transactions to trigger a snapshot to disk
"heartbeat-interval": "100" # Time (in milliseconds) of a heartbeat interval
"election-timeout": "1000" # Time (in milliseconds) for an election to timeout. See tuning documentation for details
"max-snapshots": "5" # Maximum number of snapshot files to retain (0 is unlimited)
"max-wals": "5" # Maximum number of wal files to retain (0 is unlimited)
"quota-backend-bytes": "0" # Raise alarms when backend size exceeds the given quota (0 defaults to low space quota)
"logger": "zap" # Specify ‘zap’ for structured logging or ‘capnslog’.
"log-outputs": "systemd/journal" # Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd
"enable-v2": "true" # enable v2 API to stay compatible with previous etcd 3.3.x (needed for flannel e.g.)
"discovery-srv": "" # Discovery domain to enable DNS SRV discovery, leave empty to disable. If set, will override initial-cluster.
# Certificate authority and certificate files for etcd
etcd_certificates:
- ca-etcd.pem # certificate authority file
- ca-etcd-key.pem # certificate authority key file
- cert-etcd-peer.pem # peer TLS cert file
- cert-etcd-peer-key.pem # peer TLS key file
- cert-etcd-server.pem # server TLS cert file
- cert-etcd-server-key.pem # server TLS key file
The etcd default flags/settings defined in etcd_settings can be overridden by defining a variable called etcd_settings_user. You can also add additional settings by using this variable. E.g. to override the default value for log-outputs setting add a new setting like heartbeat-interval and election-timeout add the following settings to group_vars/k8s_etcd.yml e.g.:
etcd_settings_user:
"log-outputs": "stdout"
"heartbeat-interval": "250"
"election-timeout": "2500"
The last two settings are definitely something to checkout. The values for these two settings made very much sense in my setup. The default ones were a little bit too low. I also add "peer-cert-allowed-cn": ""etcd" in my setup. The value of this variable is taken from etcd_peer_csr_cn variable (see previous blog post). When peer-cert-allowed-cn flag is specified, an etcd node can only join with matching common name (cn) even with shared CAs (certificate authority). This might be important if you have multiple etcd clusters running all with the same CA. This avoids that an etcd member joins accidentally the wrong cluster.
And then there are two more default variables which I’ll change:
etcd_interface: "{{ k8s_interface }}"
etcd_ca_conf_directory: "{{ k8s_ca_conf_directory }}"
The role will search for the certificates I created in certificate authority in the directory specified in etcd_ca_conf_directory (which in my case is just the value of k8s_ca_conf_directory defined in group_vars/all.yml) on the host where the Ansible Controller is locacted. And make sure to change etcd_interface to wg0 (or the variable k8s_interface defined in group_vars/all.yml) instead of tap0 if you followed my blog series so far and used WireGuard VPN and the interface wg0. That’s important as the etcd cluster nodes and the kube-apiserver must be able to talk to each other!
And finally the firewall rules must be adjusted. So far I only allowed port 22222 (for SSH) and 51820 (for WireGuard). For etcd port 2380 (default) needs to be open between the etcd cluster members for cluster/peer communication and normally port 2379 needs to be open for etcd to kube-apiserver communication. But as I’ll use my etcd cluster also for Cilium and Traefik as mentioned in the previous blog post I need to allow port 2379 for the whole cluster. So I need to extend harden_linux_ufw_rules in group_vars/k8s_etcd.yml accordingly and the final list will look like this:
harden_linux_ufw_rules:
- rule: "allow"
to_port: "22222"
protocol: "tcp"
- rule: "allow"
to_port: "51820"
protocol: "udp"
- rule: "allow"
to_port: "2379"
protocol: "tcp"
from_ip: "10.0.11.0/24" # K8s Controller + Worker
- rule: "allow"
to_port: "2380"
protocol: "tcp"
from_ip: "10.0.11.2/32" # etcd node1
- rule: "allow"
to_port: "2380"
protocol: "tcp"
from_ip: "10.0.11.5/32" # etcd node2
- rule: "allow"
to_port: "2380"
protocol: "tcp"
from_ip: "10.0.11.8/32" # etcd node3
To deploy the firewall changes I re-deploy my harden-linux role:
ansible-playbook --tags=role-harden-linux k8s.yml
Now I can deploy the etcd role:
ansible-playbook --tags=role-etcd k8s.yml
This will install the etcd cluster and start the etcd daemons. Have a look at the logs of your etcd hosts if everything worked and the etcd nodes are connected. Use journalctl --no-pager or journalctl -f or journalctl -t etcd to check the systemd log. E.g.
ansible -m command -a "systemctl status etcd" k8s_etcd
ansible -m command -a "journalctl --boot -t etcd" k8s_etcd
Afterwards we can use Ansible to check the cluster status e.g.:
ansible -m shell -e "etcd_conf_dir=/etc/etcd" -a 'ETCDCTL_API=3 etcdctl endpoint health \
--endpoints=https://{{ ansible_wg0.ipv4.address }}:2379 \
--cacert={{ etcd_conf_dir }}/ca-etcd.pem \
--cert={{ etcd_conf_dir }}/cert-etcd-server.pem \
--key={{ etcd_conf_dir }}/cert-etcd-server-key.pem' \
k8s_etcd
I use Ansible’s shell module here. I also set a variable etcd_conf_dir which points to the directory where the etcd certificate files are located. That should be the same value as the value of etcd_conf_dir variable of the etcd role. Since my etcd processes listen on the WireGuard interface I use ansible_wg0.ipv4.address here as wg0 is the name of my Wireguard interface. If you use a different port than 2379 then of course you need to change that one too. You should see now a output similar to this:
etcd-node1 | CHANGED | rc=0 >>
https://10.8.0.101:2379 is healthy: successfully committed proposal: took = 2.807665ms
etcd-node2 | CHANGED | rc=0 >>
https://10.8.0.103:2379 is healthy: successfully committed proposal: took = 2.682864ms
etcd-node3 | CHANGED | rc=0 >>
https://10.8.0.102:2379 is healthy: successfully committed proposal: took = 10.169332ms
Or:
ansible -m shell -e "etcd_conf_dir=/etc/etcd" -a 'ETCDCTL_API=3 etcdctl --write-out=table endpoint status \
--endpoints=https://{{ ansible_wg0.ipv4.address }}:2379 \
--cacert={{ etcd_conf_dir }}/ca-etcd.pem \
--cert={{ etcd_conf_dir }}/cert-etcd-server.pem \
--key={{ etcd_conf_dir }}/cert-etcd-server-key.pem' \
k8s_etcd
which will show you a table like this:
k8s-010101.i.example.com | CHANGED | rc=0 >>
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.11.2:2379 | 296f59da7313dfd7 | 3.5.9 | 20 kB | false | false | 2 | 11 | 11 | |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
k8s-010201.i.example.com | CHANGED | rc=0 >>
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.11.5:2379 | 4a37ed13f1667e6b | 3.5.9 | 20 kB | true | false | 2 | 11 | 11 | |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
k8s-010301.i.example.com | CHANGED | rc=0 >>
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.11.8:2379 | d20e0decc9083d25 | 3.5.9 | 20 kB | false | false | 2 | 11 | 11 | |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
That cluster is in good shape. All nodes have the the same index and etcd #2 is the leader.
Next we’ll install the Kubernetes control plane.