Kubernetes Network Policies: A Practitioner's Guide

by Levent Ogut

Providing security for our infrastructure and applications is a never-ending continuous process. This article will talk about security in Kubernetes clusters, traffic incoming and outgoing to/from the cluster, and the traffic within the cluster. Some organizations behave as if their own workloads can be malicious and design their security policies accordingly. In addition, in today's world, we all use third-party plugins, libraries, and pieces of code from external resources. Although this has been increasing productivity, this also brings many security concerns. Isolating the traffic incoming and outgoing to our applications to only what’s absolutely necessary is one of the best approaches there is.

Why We Need Network Policies

It is of paramount importance to secure the traffic in our clusters. By default, all pods can talk to all pods with no restriction. NetworkPolicy resource allows us to restrict the ingress and egress traffic to/from pods. For example, it provides the means to restrict the ingress traffic of a database pod to only backend pods, and it can restrict ingress traffic of a backend pod's traffic to a frontend application only. This way, we can secure our resources so that only legitimate traffic is allowed to/from the applications. An example would be limiting traffic so that our frontend pods can only connect to the backend application, so that an attacker who compromises the front end can’t directly access the database or any other pods.

The functionality of controlling traffic is typically achieved in networks by using firewalls (software or hardware). Here in Kubernetes that functionality is implemented by the network plugins and controlled by network policies. Note that network policies are not a replacement for firewalls.

Requirements for Implementing Network Policies

Kubernetes provides networking functionality by using network plugins. Unless you have a network plugin that can implement network policies, you will not be able to use the functionality. Please note that even if the API server accepts the network policy configuration, this doesn't mean that it will be implemented unless a controller understands and implements the policy. Several network plugins support network policies and much more.

Network Plugins

There are two types of network plugins:

CNI
Kubenet

CNI type plugins follow the Container Network Interface spec and are used by the community to create advanced featured plugins. On the other hand, Kubenet utilizes bridge and host-local CNI plugins and has basic features.

Several network plugins were developed from various organizations, including but not limited to Calico, Cilium, and Kube-Router. A complete list can be found in Cluster Networking documentation. These network plugins provide Network Policy implementation and more, such as advanced monitoring, L7 filtering, integration to cloud networks, etc.

While some network plugins use Netfilter/iptables in their underlying infrastructure, others use eBPF technology on the underlying data path. Netfilter/iptables is very mature and builtin into the kernel. On the other hand, eBPF allows you to change the functionality on the fly without kernel upgrade. Not being dependent on the kernel version has led some big players to use eBPF based network plugins on very large scales.

It is imperative to select the correct network plugin for your Kubernetes cluster(s). If you are using cloud providers for your Kubernetes setup (such as AWS, Azure, GCP), they might already have deployed a network plugin that supports network policies. Please check the cloud provider documentation for further details.

Writing & Applying Network Policies

Isolation

In a Kubernetes cluster, by default, all pods are non-isolated, meaning all ingress and egress traffic is allowed. Once a network policy is applied and has a matching selector, the pod becomes isolated, meaning the pod will reject all traffic that is not permitted by the aggregate of the network policies applied. The order of the policies is not important; an aggregate of the policies is applied.

Network Policy Resource Fields

Fields to define when writing network policies:

podSelector

podSelector selects a group of pods using a LabelSelector. If empty, it would select all pods in the namespace, so beware when using it.

policyTypes

policyTypes lists the type of rules that network policy includes. Value can be ingress, egress, or both.

ingress

ingress defines the rules that will be applied to the ingress traffic of the selected pod(s). If it is empty, it matches all the ingress traffic. If it is absent, it doesn't affect ingress traffic.

egress

egress defines the rules that will be applied to the egress traffic of the selected pod(s). If it is empty, it matches all the egress traffic. If it is absent, it doesn't affect egress traffic.

Egress Rules

An array of rules that would be applied to the traffic going out of the pod. It is defined with the following fields.

Fields:

ports: an array of NetworkPolicyPort (port, endport, protocol)
to: an array of NetworkPolicyPeer (ipBlock, namespaceSelector, podSelector)

Ingress Rules

An array of rules that would be applied to the traffic coming into the pod. It is defined with the following fields.

Fields:

from: an array of NetworkPolicyPeer (ipBlock, namespaceSelector, podSelector)
ports: an array of NetworkPolicyPort (port, endport, protocol)

Walkthrough

Let's do a walkthrough of a network policy defined as below.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy-walkthrough-db
spec:
  podSelector:
    matchLabels:
      component: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 192.168.1.2/32
    - namespaceSelector:
        matchLabels:
          team: dba
    - podSelector:
        matchLabels:
          component: backend
    ports:
    - protocol: TCP
      port: 5432

This rule applies to all pods that have labels with component key and database value (component=database). Network policy affects only ingress traffic as defined in policyTypes\.

The tree ingress rule entries are evaluated with OR. Let's look at how Kubernetes interpreted the configuration using describe\ subcommand:

$ kubectl describe networkpolicy network-policy-walkthrough-db

Name:         network-policy-walkthrough-db
Namespace:    default
Created on:   2021-08-30 18:06:48 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     component=database
  Allowing ingress traffic:
    To Port: 5432/TCP
    From:
      IPBlock:
        CIDR: 192.168.1.2/32
        Except: 
    From:
      NamespaceSelector: team=dba
    From:
      PodSelector: component=backend
  Not affecting egress traffic
  Policy Types: Ingress

Host with IP "192.168.1.2", all pods in a namespace that have team label set to dba and all pods in the same namespace that has label component set to backend are allowed to reach on port 5432.

Examples

Default Deny Ingress

An all deny ingress rule with an empty podSelector (selecting all pods in the namespace) is a good starting point for a fresh cluster. You can then explicitly allow required traffic. As the podSelector is empty, it will continue to match new pods when they arrive.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress-policy
spec:
  podSelector: {}
  policyTypes:
  - Ingress

$ kubectl describe networkpolicies default-deny-ingress-policy

Name:         default-deny-ingress-policy
Namespace:    default
Created on:   2021-08-28 16:47:33 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Not affecting egress traffic
  Policy Types: Ingress

As you can see, Kubernetes interpreted our configuration as intended. All pods in the namespace are now isolated, no ingress traffic is allowed to the pods, and egress traffic is not affected.

Allow Access to a Group of Pods from Another Namespace

In this example, we will look at a network policy that allows debugging pods to connect to the application pods.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-debug
spec:
  podSelector:
    matchLabels:
      component: app
  ingress: 
  - from:
    - podSelector:
        matchLabels:
          component: debug
      namespaceSelector:
        matchLabels:
          space: monitoring
  policyTypes:
  - Ingress

Please note that here we have a single from rule. If we had put the namespaceSelector into its own rule, the meaning would change drastically; this is where podSelector and namespaceSelector are used together.

Let's check how Kubernetes interpreted the policy:

$ kubectl describe networkpolicy allow-debug

Name:         allow-debug
Namespace:    default
Created on:   2021-08-30 22:36:48 +0200 CEST
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     component=app
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      NamespaceSelector: space=monitoring
      PodSelector: component=debug
  Not affecting egress traffic
  Policy Types: Ingress

Here we only allow ingress traffic from pods with a label component set to debug in the namespaces with the label space set to monitoring.

Monitoring Network Policies

Monitoring the network policies and their behavior is an essential part of the deployment. Kubernetes offers the kubectl describe networkpolicy <NETWORK_POLICY_NAME> command to see how Kubernetes interpreted the network policy configuration. For detailed analysis, check out the network plugin's tools. Here we have a Kubernetes cluster with Cilium network plugin. Cilium offers a CLI tool, and from there, we can monitor the packets.

Let's get the IP address of our pod:

$ kubectl get pods -o wide

NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
nginx-deployment-66b6c48dd5-frsv9   1/1     Running   0          24m   10.0.0.136   valhalla   <none>           <none>

Let's get the endpoint id (in Cilium) of the pod:

$ cilium endpoint list

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])  IPv6   IPv4         STATUS    ENFORCEMENT        ENFORCEMENT                                                                                                                   
5          Enabled            Disabled          50873      k8s:app=nginx                       10.0.0.136   ready   
...

We will monitor all traffic that goes to and comes from the endpoint with id 5.

Press Ctrl-C to quit

level=info msg="Initializing dissection cache..." subsys=monitor
Policy verdict log: flow 0xf9da54c5 local EP ID 5, remote ID 1, dst port 80, proto 6, ingress true, action allow, match L3-Only, 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state new ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp SYN, ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp ACK
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK, FIN
-> stack flow 0xbbd5210b identity 50873->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.0.0.136:80 -> 10.0.0.147:39772 tcp ACK, FIN
-> endpoint 5 flow 0xf9da54c5 identity 1->50873 state established ifindex lxc4eced79e6ca0 orig-ip 10.0.0.147: 10.0.0.147:39772 -> 10.0.0.136:80 tcp ACK

We can see the traffic destined to our NGINX pod.

Policy verdict log: flow 0xf9da54c5 local EP ID 5, remote ID 1, dst port 80, proto 6, ingress true, action allow, match L3-Only, 10.0.0.147:39772 -> 10.0.0.136:80 tcp SYN

We can see the policy evaluation here.

Conclusion

We have explored why and how network policies are used within a Kubernetes cluster. Allowing only the required traffic is a security best practice, and Kubernetes allows us to implement this via declarative configuration of network policies. Since network policies depend heavily on the labels of pods/namespaces, it is straightforward to deploy rules that would also capture newly created resources.

It is highly recommended to test the network policies before applying them.

Observing traffic sources, destinations, and flows is imperative; as Kubernetes API does not include statistics, learning how to use the monitoring/troubleshooting tools of the network plugin becomes very important.

Folks at Cilium also developed a great UI Network Policy editor; make sure to check it out.

Blog