I wrote about personal highlights from KubeCon Europe 2021 yesterday and the day before. I think it is only right if I write about it a third time. For those that are unaware, I’m writing about my experience in KubeCon Europe 2021 and today is Day 3. These will be a series of notes written as a stream of consciousness without much editing. This year’s edition of KubeCon Europe was once again completely virtual.

Sessions

I started my day with some leftover sessions from yesterday. In this case, I started with Zero Pain Microservice Development and Deployment with Dapr and KubeVela by Hongchao Deng which went about presenting a solution for improving workload deployments with KubeVela and Dapr as sidecar for metrics. Two things were noticeable for me in this session: more and more tools being built to ease and abstract away the complexity of Kubernetes, in this case, KubeVela helps Platform Teams create “application-centric” resources in Kubernetes; and, once again, there’s this trend of using sidecars for deployment auxiliary technology with their applications.

In the same vein, Turning Your Cloud Native Apps Inside Out With a Service Mesh by Adam Zwickey and Liam White presented another use case of using Envoy as a sidecar to transparently create service meshes that can be managed by a Platform Team, leaving developers to focus exclusively on their own services. Have I said that there’s a trend to use sidecars with auxiliary technology? Envoy seems to be front and center in this. Above we just saw using Dapr for metrics, for example.

One of the things I enjoy about these conferences are the introductory sessions such as Introduction and Deep Dive Into Containerd by Kohei Tokunaga and Akihiro Suda where I have learned a bit more about how containerd internals function - for example, apparently you can easily build containerd clients - or CoreDNS Deep Dive: Building Custom Plugins by Yong Tang where I have understood that we can build plugins for CoreDNS, which I had no idea about. It is a great way of getting a quick introduction to a specific technology or learning about extensibility in a project.

Why Use Managed Kubernetes?: It’s Dangerous to Go Alone! by Seth McCombs is a candidate for my favourite session in this year’s KubeCon. Seth tried to dismantle the notion that you frequently hear at these conferences about “the right way” to do things. For most use cases, using managed Kubernetes services is aptly justified because, by having a team of experts in some cloud provider working day and night on these services, they can guarantee much more stability and reliance than we’d otherwise be able to. It is much more important to focus on building cool things, and features that can contribute to the core business rather than wasting time managing Kubernetes. There are cases where running your own Kubernetes is justified, particularly if you need customizations, but managed services usually contribute to fewer costs, more reliability and more time to build features. End users don’t necessarily care about whether you are running Kubernetes or using AWS/GCP. Another thing that I noticed, and I was talking about in yesterday’s updates, was Seth’s reluctance in managing your own etcd - it truly must be a pain. End users don’t necessarily care about running on Kubernetes. Seth ended the session with a great quote: “Build cool stuff. In whatever way works for you. With the tools that work for you”. Kubernetes should be just one more tool in your toolbox.

I also had a chance to watch three good sessions on security topics. Uncovering a Sophisticated Kubernetes Attack in Real-Time by Jed Salazar and Natália Réka Ivánkó where they suggested that security could be more like SRE. They have showcased a solution that leverages https://ebpf.io/ and https://cilium.io/ for security monitoring in Kubernetes (again, with sidecars in pods!!). This solution allows to proactively acting whenever specific actions are executed in Kubernetes clusters, moving security from something that is always on fire to being something proactive and monitored in real-time. Hacking into Kubernetes Security for Beginners by Ellen Körbes and Tabitha Sable was an extremely creative session where I’ve heard things such as: “We are made of stars but your RBAC shouldn’t be.” More than being simply creative, these sessions expanded my understanding of the attack surface in workloads running in Kubernetes. Misconfigured RBAC, powerful tools configured and running pods, potential vulnerabilities in Kubernetes APIs, image vulnerabilities. There’s so much to take care of! A few suggestions have been left about having audit logs in Kubernetes APIs, leveraging admission controllers and CI scanning for vulnerabilities. Finally, The Art of Hiding Yourself by Lorenzo Fontana presented security issues in Kubernetes from the perspective of someone trying to hide inside a given system. It talks about ways to hide process activity, network activity and storage activity, and even pods!! What fascinated me the most was how fairly trivial it is to hide these things, as long as you know how to circumvent the common ways of monitoring these things. Sometimes you need to look at things from a different angle to be able to see clearly. This session leveraged Falco for real-time security monitoring in Kubernetes.

Finally, one of the sessions I was most excited about was Sidecars at Netflix: From Basic Injection to First Class Citizens by Rodrigo Campos Catelin and Manas Alekar. I am a big fan of Netflix’s engineering culture which prompted my enthusiasm. This is a story of how Netflix evolved from an architecture where EC2 VMs stored application code, along with auxiliary technology for logging, metrics and networking, through a solution with Titus where each EC2 VM was running everything in containers, and now with Kubernetes and trying to advance Kubernetes sidecars. They have laid out some sidecar issues that they are currently facing such as no startup order guarantees, no shutdown order guarantees, can’t be straightforwardly used with Jobs and can’t use them with initContainers. Interestingly enough, the thing I was most fascinated by was KEP 753. I had no idea but it seems that there have been some proposals to advance sidecars to First Class Citizens. The proposal didn’t go anywhere but there are still “pre-proposal” efforts to start a new proposal and introduce sidecars in Kubernets’ specification. I was amazed because there are so many instances of companies using sidecars in production, I’ve mentioned a few of them in the last couple of days, but there’s no real standard/specification around sidecars in Kubernetes.

Cool Tech

• Crossplane: “Compose cloud infrastructure and services into custom platform APIs”.
• KubeVela: “Make shipping applications more enjoyable”.
• OpenKruise: “A Kubernetes extended suite for application automations”.
• Dapr: “Simplify cloud-native application development”.
• kubespray: “Deploy a Production Ready Kubernetes Cluster”.
• nerdctl: “Docker-compatible CLI for containerd, with support for Compose”.
• eBPF: “eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules”.
• Cilium: “eBPF-based Networking, Observability, and Security”.
• yugabyteDB: “Open source, cloud native relational database for powering global, internet-scale apps”.
• Titus: “a container management platform that provides scalable and reliable container execution and cloud-native integration with Amazon AWS”.