When I started using Kubernetes, and consequently kubectl, figuring out how to quickly view my pod's logs was always a pain point. Why was the verb/noun usage not the same? How do I know if there's more than one container running in the pod? How do I filter any of this except with CLI magic like grep or awk? In this post, I'm hoping to shed some light on the multitude of ways you can grab logs locally from your Kubernetes cluster.

If you don't have a Kubernetes cluster accessible, feel free to use any of these:

• https://kind.sigs.k8s.io/
• https://kubernetes.io/docs/tasks/tools/install-minikube/
• https://github.com/rancher/k3d

kubectl logs

First up is the simplest of all the logging options: kubectl logs

Here is some yaml below that will generate the date and time every second. If you have only a single container in the pod, you can simply run kubectl logs echo-date to see all of the output.

apiVersion: v1
kind: Pod
metadata:
  name: print-date
  namespace: default
spec:
  containers:
    - name: main
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done

Now, what happens when you add a second image into the pod? Let's apply the same yaml with a second image added into the pod.

apiVersion: v1
kind: Pod
metadata:
  name: print-dates
  namespace: default
spec:
  containers:
    - name: main
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done
    - name: sidecar
      image: alpine
      args:
        - /bin/sh
        - -c
        - >
          while true;
          do
            date
            sleep 1
          done

When there are multiple containers running inside a single pod, you must pick the container. In order to see how many containers are in the pod, you can run kubectl get pods. Below, the READY field indicates how many containers are up and running in the pod.

NAME          READY   STATUS    RESTARTS   AGE
print-date    1/1     Running   0          5m25s
print-dates   2/2     Running   0          5m56s

Now, you can get the logs for either container: kubectl logs print-dates sidecar. And in case you're working with a pod whose containers' names you don't know, here is a fancy command utilizing json parsing: kubectl get pod print-dates -o json | jq .spec.containers[].name

Alright, so we've got some basic knowledge on how to see logs,
but I'd really like to filter these logs down some and see them streaming live to the terminal. For now, let's switch back to the single container pod in order to make the commands a bit easier.

First up, let's follow the logs live: kubectl logs print-date --follow

Next, let's filter based on the timestamp: kubectl logs print-date --since=10s

I want to follow this info without all the previous logs drowning my terminal in rows of data though: kubectl logs print-date --tail=1 --follow.

So, it's pretty evident kubectl logs provides a lot of extensibility on how to view container logs, but let's take this further and look at some community built options.

stern

stern is a popular open source option for viewing pod logs by name, namespace, time, and many other options. You can think of it as a kubectl logs aggregator. It's a great solution when trying to debug a solution that may span multiple pods across a namespace.

To install on Mac: brew install stern

Because we have two pods populating logs in the default namespace, let's start with viewing both of these at the same time: stern -n default print-. By adding print-, we are limiting the logs to only pods that are prepended with that pod name. One of the great things about stern is that is classifies each pod's logs as a different color allowing easy distinction between the two. This is, perhaps, the best feature I've come across when debugging.

While this output is not color coded in the markdown below, it is worth noting that the pod and container are added into the logs:

print-date main Thu Oct  8 04:06:52 UTC 2020
print-dates main Thu Oct  8 04:06:52 UTC 2020
print-dates sidecar Thu Oct  8 04:06:53 UTC 2020
print-date main Thu Oct  8 04:06:53 UTC 2020

stern --help provides tons of really good options for filtering pods, so I won't go too in depth with examples. However, I'd like to show an example above for a similar solution of printing since a certain time. Except we'll be printing output for multiple containers now.

• Print by namespace: stern -n default .
• Print all pods: stern .
• Print the last 15 seconds for both pods' logs: stern -t --since 15s print-date

Definitely checked out the README for additional examples as well.

kubectl plugins

For the final iteration, let's take a look at kubectl plugins, which are basically just binaries prepended with kubectl-. Krew is a package manager that allows easy installation of these. We can use kubectl tail to accomplish much of the same logging capabilities as stern.

While this can be installed by kubectl krew install tail, the GitHub repo resides here.

In order to compare to stern, here are basically the same examples:

• Print by namespace: kubectl tail --ns default
• Print all pods: kubectl tail
• Print the last 15 seconds for both pods' logs: kubectl tail --pod print-date --pod print-dates --since 15s

Make sure to check out the flags with kubectl tail --help to see all potential options!

wrap up

While none of this was exactly groundbreaking information, I hope it created some muscle memory of using kubectl logs and showed open source solutions that are a great alternative as you become a more advanced k8s user. Happy programming!

p.s. -- it's pronounced kube-see-tee-el :p