A matter of trust

Many interesting software are coming from the community, many are distributed through the package manager of the operating system. But for the others, you can download them from Github release pages, use snap or homebrew to cite a few. But this last installation method bypasses the security team that tries to improve the security of your operating system. By doing so, you are implicitly trusting the author he is not distributing malware or implementing backdoors. How many tools did you install by hand? Do you really trust all of them? Confidence is very important, yet it would be nice to limit capabilities for a set of binary that you don't fully trust. In this blog post, we will use Tetragon to forbid network usage for tools that don't need to.

Goal

We will separate tools installed locally into two families. On one side, tools that need network access in a specific directory. Another directory for tools that don't need internet access.

For example the following tools use network sockets:

• k9s
• kubectl
• helm

while these do not:

• jq
• yq
• jless

We will move the last tools in a specific directory ~/bin-no-network/ and use Tetragon to inject a policy in the kernel as a eBPF program to kill any binary located in ~/bin-no-network/ trying to open a network socket.

Tetragon installation

Tetragon is "Kubernetes-aware" but it can also be used outside Kubernetes on a regular workstation. You can deploy Tetragon as a container:

$ docker run --name tetragon --rm -d                 \
    --pid=host --cgroupns=host --privileged          \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
    quay.io/cilium/tetragon:v0.10.0

Because Tetragon needs to inject code in the kernel, we need to bypass most of the docker isolation mechanism. We only use the packaging feature of docker to avoid installation of system libraries and the binary. So you will have to trust ;-) this tetragon binary because it will run like a process running as root on your workstation. Note the mount point: /sys/kernel/btf/vmlinux, this is a kind of bridge to eBPF kernel features.

eBPF Principle

When using eBPF, applications are always composed of two parts, one deployed in the Kernel as an eBPF program and another running as a regular program in "user space". The user space program (Tetragon) will inject a small eBPF program in the kernel to intercept some system calls. This means that every application that uses this system call will trigger this eBPF program that can observe or modify the system call result. A specific data structure is then created in the kernel: a ring buffer. This is used by the eBPF program to store some interesting data. Finally, the user space program Tetragon, which has also read-only access to this data structure, will be able to retrieve information gathered by the eBPF program.
One good point with this architecture is that evaluation of rules is done within the kernel and does not require communication with the user space program. The only drawback is that information observed by the eBPF program can be overridden by new incoming information if the user space part does not read information fast enough. This is how ring buffers are designed.

Writing a Policy

Our goal is to forbid network usage coming from binaries in a specific folder.
For this we will need a CLI to interact with tetragon. We can use the tetra binary in the docker container:

$ alias tetra="docker exec -ti tetragon tetra"
$ tetra version
server version: v0.10.0
cli version: v0.10.0

Even if we are not using Kubernetes, we still need to write some yaml as Tetragon only understands TracingPolicy objects. A TracingPolicy is a kubernetes custom resource to install hooks in the kernel and actions.

Let's start with a TracingPolicy from the Tetragon documentation:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int

This Tracing policy will just observe network events:

• socket creation (tcp_connect)
• traffic in the socket (tcp_sendmsg)
• socket close (tcp_close)

Now we need to add:

• an action when this kind of event is detected
• a filter to only apply this action if this is generated from a binary located in our specific directory ~/bin-no-network/.

We need to add a selectors section in the TracingPolicy and use the matchBinaries selector to apply this policy only for binary in the ~/bin-no-network/ folder:

    selectors:
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Unfortunately, only 'In' and 'NotIn' operator are implemented for the matchBinaries selector. We cannot use the 'Prefix' operator like this:

    selectors:
    - matchBinaries:
      - operator: "Prefix"
        values:
          - "/home/jacroute/bin-no-network/"

So the TracingPolicy should look like:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector

Deploy the TracingPolicy

We first need to transfer the file to the container. Suppose that you stored the TracingPolicy in bin-no-network.yaml file, you can transfer the policy using the docker cp command:

$ docker cp bin-no-network.yaml tetragon:/tmp/bin-no-network.yaml

Then we can use the tetra cli to deploy this policy:

tetra tracingpolicy add /tmp/bin-no-network.yaml

Testing the TracingPolicy

Now we can test if the policy is blocking network access for the two binaries listed in the policy.

$ mkdir ~/bin-no-network/
$ cp /usr/bin/curl ~/bin-no-network/
$ cp /usr/bin/jq ~/bin-no-network/
$ ~/bin-no-network/curl google.fr
[1] 122677 killed ~/bin-no-network/curl google.fr
$ echo "{}" | ~/bin-no-network/jq
{}

curl tried to open a socket but the process was killed. jq does not try this kind of system call and was not killed.

Automatically populating the Policy

We can build a shell script to maintain the list of binaries synchronized with the content of the ~/bin-no-network/ directory. Using find we can list binaries in this folder:

$ find ~/bin-no-network/ -executable -type f
/home/jacroute/bin-no-network/curl
/home/jacroute/bin-no-network/jq

Then with awk with can add surrounding spaces and quotes needed to integrate the yaml:

find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}'
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Finally, integrate this in the yaml and inject the policy in the kernel with the following script:

#!/bin/bash

docker exec tetragon tetra sensors rm connect

policy=$(mktemp -p /dev/shm)

cat << EOF > $policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
$(find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}')
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector
EOF

docker cp $policy tetragon:/tmp/policy.yaml
docker exec tetragon tetra tracingpolicy add /tmp/policy.yaml
rm $policy

Building trust in your relationship with your workstation XD

After a first date with Tetragon during the Kubernetes Community Days during 2023, I was seduced by the way it's implemented: most of the time new features are implemented on top of others creating a complex multi layered cathedral. Perfect for a wedding, but I prefer simplicity.

Using eBPF to enforce "security" seems to be very efficient from the performance point of view. Bypassing Tetragon security by denial of service attacks is unlikely. For example, the sigkill action triggered by the Tetragon’s eBPF program is done synchronously, in the kernel and not at user space, which makes it hard to bypass. In other words, the security mechanism is fully and autonomously implemented at kernel level.

This mechanism proves to be effective in blocking network activity and building trust in our beloved workstation even when running potentially malicious binaries.

Using Tetragon in a Kubernetes context is the next step. The plan is to observe the behavior of an application in a development environment (files read/write, command executed) and generate a profile. Then, promote this profile from development to production, and enforce an allow-only behavior using Tetragon.