TauCeti.blog
Posts Tags
TauCeti.blog

Kubernetes the Not So Hard Way With Ansible - Network policies with kube-router

     1341 words  7 minutes 

2020-08-31

While parts of this blog post are still valid (and Kube-router is still a very nice and up2date Kubernetes networking solution) I switched to Cilium as my “all in one” networking solution as described in Kubernetes the not so hard way with Ansible - The worker. Besides network policies this thing just has everything ever needed for Kubernetes networking and still it’s easy to use. So this blog post is just left for reference reasons. It won’t be updated anymore.

If you followed my Kubernetes the Not So Hard Way With Ansible blog posts so far you should now have a pretty usable Kubernetes cluster running. What’s missing - but not strictly required - besides persistent storage (like GlusterFS) are Network Policies. From the Kubernetes documentation: A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. NetworkPolicy resources use labels to select pods and define rules which specify what traffic is allowed to the selected pods.

As we already have Kubernetes networking up and running we only need a network plugin that implements the network policy specification. Calico is one of the different options but it’s a bit to heavy for what we need for our setup. CloudNative Labs kube-router provides a network policy implementation and can be activated basically with one switch. kube-router provides many more features like distributed load balancing and routing for pods but we will only use the firewall part which implement’s ingress (incoming traffic to pods) and egress (outgoing traffic from pods) firewall support. The nice thing here is that it only uses native Linux networking tools to accomplish this. So it makes it quite “easy” (well say it’s at least possible if you deep dive into Linux netfilter/iptables ;-) ) to debug if something strange happens. There is more information in Kube-router: Enforcing Kubernetes network policies with iptables and ipset. So let’s start ;-)

We basically need quite a few Kubernetes resources that we already used for Traefik ingress. The first resource is a ClusterRole. kube-router needs some cluster wide permissions and we define this permissions in this ClusterRole. So lets create a variable kube_router_clusterrole in Ansible’s group_vars/all.yaml variable file (or where it fits best for you):

kube_router_clusterrole: |
  ---
  kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: kube-router
  rules:
    - apiGroups: [""]
      resources:
        - namespaces
        - pods
        - services
        - nodes
        - endpoints
      verbs:
        - get
        - list
        - watch
    - apiGroups: ["networking.k8s.io"]
      resources:
        - networkpolicies
      verbs:
        - get
        - list
        - watch
    - apiGroups: ["extensions"]
      resources:
        - networkpolicies
      verbs:
        - get
        - list
        - watch

As you can see here we create a ClusterRole called kube-router. We specify no namespace as ClusterRole’s are not namespaced. There’re resources and verbs defined for three apiGroups. The fist one indicates the core API group as it’s value is "". For all three apiGroups we want to allow get,list and watch (the verbs) for the resources defined.

Next we define a ClusterRoleBinding:

kube_router_clusterrolebinding: |
  ---
  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: kube-router
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: kube-router
  subjects:
  - kind: ServiceAccount
    name: kube-router
    namespace: kube-system

The name of the ClusterRoleBinding is kube-router. It’s basically the glue between the ClusterRole and the ServiceAccount (see below). This means in subjects we list users, groups or service accounts and in roleRef we assign basically what we want to allow. In the case above we allow everything defined in our kube-router ClusterRole.

I already mentioned the service account so let’s add a Ansible variable for it:

kube_router_serviceaccount: |
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: kube-router
    namespace: kube-system

Only a few lines here. Later when we configure the kube-router DaemonSet we configure it to use this service account which enables kube-router to get all the information from the API server it needs to work properly. E.g. if new pods are created then kube-router gets the information (via so called watcher) that new pods are created and applies the network policy accordingly if needed (that’s the job of the different controller that kube-router provides depending what resource needs to be managed).

And while talking about the DaemonSet… ;-) Here is the Ansible variable for it:

kube_router_daemonset: |
  ---
  kind: DaemonSet
  apiVersion: apps/v1
  metadata:
    name: kube-router
    namespace: kube-system
    labels:
      k8s-app: kube-router
  spec:
    template:
      metadata:
        labels:
          k8s-app: kube-router
        annotations:
          scheduler.alpha.kubernetes.io/critical-pod: ''
      spec:
        serviceAccountName: kube-router
        hostNetwork: true
        containers:
        - name: kube-router
          image: cloudnativelabs/kube-router:v0.0.20
          args: 
          - --run-router=false
          - --run-firewall=true
          - --run-service-proxy=false
          - --masquerade-all
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: lib-modules
            mountPath: /lib/modules
            readOnly: true
          - name: cni-conf-dir
            mountPath: /etc/cni/net.d
        tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        volumes:
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: cni-conf-dir
          hostPath:
            path: /etc/cni/net.d

A DaemonSet means that the specified container (cloudnativelabs/kube-router:v0.0.20 in our case / see https://github.com/cloudnativelabs/kube-router/releases for the latest releases) will run on every worker node. That’s what we need here as the firewall rules needs to be modified on this nodes accordingly to enforce the network policy.

Let’s get quickly through the important DaemonSet config settings:

  metadata:
    name: kube-router
    namespace: kube-system
    labels:
      k8s-app: kube-router

That’s pretty obvious: We name the DaemonSet kube-router, run it in kube-system namespace and put a label on it.

Next comes the specification of the template that will be used by Kubernetes to create the pods on each worker node. The important parts here:

serviceAccountName: kube-router
hostNetwork: true

This specifies that this DaemonSet should use the ServiceAccount we created above. This basically assigns kube-router the permissions which we defined in the kube-router ClusterRole. Additionally the pods should use the host’s network which is of course important as we need the firewall rules at this level and not somewhere above this “layer”.

        containers:
        - name: kube-router
          image: cloudnativelabs/kube-router:v0.0.20
          args:
          - --run-router=false
          - --run-firewall=true
          - --run-service-proxy=false
          - --masquerade-all
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent

This is basically the heart of the whole DaemonSet. This config will cause the cloudnativelabs/kube-router be pulled if it is no already on the worker node’s Docker image cache. The pods have privileged access because they need to do some “low level” tasks on the host which needs this kind of permissions. And we only turn on the firewall functionality of kube-router via --run-firewall=true argument.

As enforcing NetworkPolicy is a critical thing we can tell Kubernetes that these pods are kind of super important (see Guaranteed Scheduling For Critical Add-On Pods ). This means that Kubernetes tries it’s best to schedule such kind of pods over “normal” pods if resources are available. It will unschedule “normal” pods to get critical pods up and running. For this to work we need a few settings:

        annotations:
          scheduler.alpha.kubernetes.io/critical-pod: ''

and

        tolerations:
        - key: CriticalAddonsOnly
          operator: Exists

Now to roll out kube-router get the playbook I created by cloning my ansible-kubernetes-playbooks repository e.g.:

git clone https://github.com/githubixx/ansible-kubernetes-playbooks.git

Then cd kube-router and run the playbook with

ansible-playbook install_or_update.yml

This will install all the resources we defined above and of course the kube-router DaemonSet.

Now you should be able to play around with Network Policies. One thing to start with is block all traffic between all pods (here for the default namespace):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Save the content in default-deny.yaml e.g. and run kubectl apply -f default-deny.yaml. Afterwards pods shouldn’t be able to communicate anymore. But this will also prevent Traefik (see /posts/kubernetes-the-not-so-hard-way-with-ansible-ingress-with-traefik/) ) to send requests to e.g. some nginx pods that deliver our static blog or whatever. So with this policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: best-blog-ever-allow-external
spec:
  podSelector:
    matchLabels:
      app: best-blog-ever
  ingress:
  - from: []
    ports:
    - port: 80

we will allow requests to port 80 from any ([]) source to pods that have the label app=best-blog-ever assigned. Adjust the settings to your need, save them into best-blog-ever-allow-external.yaml and run kubectl apply -f best-blog-ever-allow-external.yaml. This will allow our Traefik ingress again to access the pods that matches the app=best-blog-ever label.

Have fun! ;-)

Updated on 2020-08-31
Back | Home