KubeCon + CloudNativeCon North America 2023 was held in Chicago from November 6-9. This was the third North American KubeCon since the start of the COVID-19 pandemic.

I wrote my first KubeCon wrapup post for KubeCon San Diego in 2019. If you've read the past wrapups, you'll know that I developed a specific style for them. I live-tweeted about the talks I attended and other happenings and then pulled in my tweets and others for the posts.

Given the weird place that X/Twitter is at and the engagement problems many people have noticed, I decided not to live tweet this time. I took notes and wrote up my thoughts afterward.

This was also my last KubeCon with Loft Labs. I've had a lot of fun talking about the company's tools (especially vCluster) with folks in the Kubernetes community, but it's time for a new challenge for me. I also really need a break. If you know someone in the cloud native space looking for a Developer Relations person, feel free to have them reach out to me as long as they can be a little patient with the start date. The best way to reach me is LinkedIn.

Also, if you'd like to view the videos for the talks I recommend they should be posted within a couple of weeks on the CNCF's YouTube channel.

Pre-Conference Activities

I spoke at KubeCon Detroit last year and at KubeCon Amsterdam in April, but I didn't have any talks accepted this time. I did present at two events before the conference proper, though.

My first talk was at Cloud Native Rejekts, one of my favorite community conferences. If you're unfamiliar with Rejekts, the idea is to give a space for people to present their ideas that weren't accepted for KubeCon. The speakers and talks are always very high quality, and a lot of my favorite people in the community show up. My talk was called Open Source Dev Containers with DevPod, and I had a lot of fun presenting it. I talked through the struggles involved with providing easy-to-use and repeatable dev environments and did a demo of DevPod.

Adrian Mout from Chainguard speaking at Cloud Native Rejekts

I also did a lightning talk about vCluster at Multi-TenancyCon, one of the KubeCon co-located events. I didn't have much time to attend the rest of the event, but multi-tenancy is a topic that really interests me. It was fun to speak at co-located events, and at some point the CNCF started giving free KubeCon registrations to the co-located event speakers, so that was nice.

The one downside of attending these events is that they make the KubeCon week much longer. In my case, the trip went from five days for the conference proper to eight days. It was worth it for me, but it's something to consider.

I also attended the Lightning talks on Monday evening, which were a lot of fun. One of my favorites was the talk about the CNCF's Deaf and Hard of Hearing Working Group. I wasn't aware of their work around making events and meetings more accessible, and it's fantastic. I also loved Tim Hockin's talk about the problems with the Service primitive in Kubernetes and the Gateway API.

This was my first time making it to the Lightning Talks, and I will be back in the future.

Day One - Tuesday

This time around, the KubeCon schedule changed from Wednesday-Friday to Tuesday-Thursday. I liked that change as it meant traveling home on Friday instead of on the weekend. Thank you CNCF for that.

Keynotes

The opening morning's keynotes focused a lot on running AI/ML workloads. I didn't sense as strong a theme at this KubeCon compared to some in the past, but if there was a central theme, that was it. My friend Joseph Sandoval did a panel on AI/ML, and Taylor Dolezal did a panel with end users.

My favorite part of these keynotes was the panel about sustainability called Environmental Sustainability in the Cloud Is Not a Mythical Creature, hosted by Frederick Kautz. This is a super important topic, and I'm always happy to hear about the advances in this area. The talk mentioned Kepler, a tool I've wanted to look at.

There were also updates from the CNCF's graduated projects, which now include Cilium and Istio.

The CNCF also put together a very nice In Memorium video with tributes to folks in the cloud native community who died this year. I was happy to see my friends Kris Nova and Carolyn Van Slyck included. They were both fantastic people who contributed to the community, and their losses were huge blows. There is a GitHub repo where you can add your memories of those folks.

A Practical Guide to eBPF Licensing: Or How I Learned to Stop Worrying and Love the GPL - Jef Spaleta & Bill Mulligan, Isovalent

I don't focus much on open source licensing, but I wanted to learn how it impacts projects using eBPF. Jeff and Bill both work at Isovalent, and they did a thorough job explaining the situation. The portions of eBPF projects that run in the kernel are required to use the GPL, but CNCF projects must use the Apache 2.0 license. The recommendation was to use the GPL for the kernel bits and Apache 2.0 for the parts of the software that run in userspace. The CNCF has made an exception for projects that use eBPF, allowing them to use the GPL for the code that runs in the kernel.

This was a situation that I wasn't aware of at all. If you are working with eBPF projects, this talk is worth watching.

When Is a Secure Connection Not Encrypted? and Other Stories - Liz Rice, Isovalent

Next, I went to another eBPF talk. If you have read my last few KubeCon wrapups, you know already that it's an interest of mine. I'm also a big fan of Liz's. We had a great conversation on my podcast Kube Cuddle last year.

This talk covered how Cilium and other service meshes handle encryption and identity. It's a bit hard for me to sum up because it was pretty technical. Networking also isn't my strongest area. But if you are interested in networking and Cilium, check this one out. Liz is a great speaker.

Demystifying Service Mesh: Separating Hype from Practicality - Brian Redmond & Ally Ford, Microsoft

This was an engaging introduction to service mesh. It focused on the features most meshes provide, like observability, tracing, traffic management, blue/green deploys, canary testing, A/B testing, and even fault injection.

You may have heard the term Progressive Delivery (coined by James Governor) used to refer to some of these practices, as well as things like feature flags. Progressive delivery practices allow teams to deploy more safely, and deployment safety is a huge factor in developing high-performing teams. If you've ever been on call, you understand. Service meshes make things like using canaries to test and having rollbacks easy to implement.

If you are new to service mesh and want to see what it can offer your team, check out this talk.

Service Mesh Battle Scars: Technology, Timing, and Tradeoffs - Keith Mattix, Microsoft; John Howard, Google; Lin Sun, solo.io; Thomas Graf, Isovalent; Flynn, Buoyant

Could I go to yet another talk about service mesh? Yes, I could. This one was very different, though. It was a panel hosted by Keith Mattix, and the panelists represented Cilium (Thomas), Istio (Lin and John), and Linkerd (Flynn).

The focus was on areas where the approaches of the tools differ, and things got a bit spicy at times (which was intended). Keith was a very entertaining moderator, and the panelists were all experts in their fields. If you are interested in the differences between these projects or like panels that get a bit contentious, this presentation is for you. I did enjoy it, and it was great to have something entertaining at the end of the day.

I decided to take it easy in the evening after the talks. This was already day five of my trip, and I was feeling it. These big conferences like KubeCon can be very draining, so I focus on pacing myself. If you are new to this kind of conference, it's important not to feel like you have to do everything possible. It's okay to take a break and recharge or to spend time on the hallway track instead of seeing a talk in every slot.

Day Two - Wednesday

Keynotes

The day two keynotes began with a talk from Hermanth Malla and Laurent Bernaille of Datadog, who talked through an incident that caused an almost 24-hour outage for Datadog. That outage would be rough for any application, but I'm sure many customers were caught without a backup method to observe their systems. I have a lot of respect for folks who will talk openly about outages like this and share learnings with the community.

Other highlights for me were a panel on inclusion and Jeremy Rickard from Microsoft talking about Long Term Support (LTS) for Kubernetes. A Kubernetes Enhancement Proposal (KEP) is open to change the supported period from 9 months to a year, which is a great idea.

The Community Awards have always been a favorite part of KubeCon for me. Those folks put a lot of time and energy into improving the community, and it's great to see them get recognized for it. You can see a list of the award winners in this CNCF blog post. Congratulations to all of them.

Winner of the 2023 Top Documentarian award, Divya Mohan

Learning Kubernetes by Chaos – Breaking a Kubernetes Cluster to Understand the Components - Ricardo Katz, VMware & Anderson Duboc, Google Cloud

This was one of my favorite talks of the conference. The premise was to fix a broken kind cluster bit by bit, and the speakers explained the different components of the cluster as they fixed them (apiserver, controller manager, scheduler, etc.). I don't want to spoil any of the jokes, so I will leave it by saying this was a brilliant combination of humor and education. I highly recommend watching it, especially to beginners.

Dungeons and Deployments: Leveling up in Kubernetes - Noah Abrahams, Oracle; Natali Vlatko, Cisco; Kat Cosgrove, Dell; Seth McCombs, AcuityMD

The other talk I saw on Wednesday was another one filled with jokes. In this one, the speakers explained some parts of Kubernetes by playing a tabletop role-playing game. There were a lot of jokes and plenty of puns that left people in the audience groaning. This one was heavier on the humor than the education, but it was a lot of fun.

Documentary Film - eBPF: Unlocking the Kernel

I was very much looking forward to the premiere of the new documentary about the creation and growth of eBPF. I loved the Kubernetes Documentary from the same filmmakers, but I knew the Kubernetes story better going in than I knew this one.

I was initially introduced to eBPF through Brendan Gregg, who was at Netflix back then. Brendan was posting on Twitter about the Linux performance flame graphs he generated with eBPF, and I saw him speak at SRECon about that topic. But it was several years later before I understood that eBPF can do much more, including networking and other observability.

I could go on a lot more about the film, but I think I will write a separate review of it soon. So, for now, I will leave off by saying that I recommend it for folks interested in eBFP and that you can watch it for free on YouTube.

After the film, I headed over with some of the Isovalent folks to their post-event party. I saw a lot of friends and had a great time. I feel fortunate that I've been able to connect with so many people in this community and learn from them, whether it's about open source tools or other aspects of what we do, like community and inclusion. Thanks to Isovalent for throwing such a fun party.

Day Three - Thursday

Keynotes

I was dragging by Thursday, day seven of my trip, so I missed the keynotes. I heard from multiple people that Tim Hockin's keynote was great, though, so I watched it afterward on the conference platform.

Tim's talk was called Kubernetes in the Second Decade. It was a very interesting look from a Kubernetes expert at what directions the project should take in the next ten years and what the challenges are. Tim covered topics like running AI/ML workloads, multi-cluster, complexity, and reliability. He introduced the concept of a complexity budget, which I loved. He said that there's a finite amount of complexity we can add to Kubernetes and that we need to say no to some things now so we can do other cool things later.

I strongly recommend watching Tim's talk when the videos are released.

Despite my final-day fatigue, I made it to a couple of sessions on Thursday.

Sidecar Containers Are Built-in to Kubernetes: What, How, and Why Now? - Todd Neal, Amazon & Sergey Kanzhelev, Google

Making sidecar containers first-class citizens is a change I wasn't aware of before I saw the KubeCon schedule. This talk explained how the sidecar containers work, and this talk should be helpful for many people as loads of us use sidecars.

The new sidecars are basically init containers that continue to run and are set to restart always. They start before the "main" container and end after it so that they will capture data like logs and metrics for the primary container’s entire lifecycle.

The feature is alpha in Kubernetes 1.28 and will be beta in 1.29. It’s super useful work from the SIG Node team. Big thanks to them.

Releasing Kubernetes and Beyond: Flexible and Fast Delivery of Packages - Grace Nguyen, University of Waterloo; Adolfo Garcia Veytia, Chainguard; John Anderson, Ditto

The final talk I saw at KubeCon Chicago was in the last slot of the day, and it was a presentation by members of the SIG Release team.

If you've read my past wrapup posts, you know that I have a lot of love for SIG Release. They do a lot for the Kubernetes project, and it's all very much in the "chop wood, carry water" vein. Release engineering tends not to get much attention until something goes wrong, and it's very challenging work. So I really appreciate the folks on this team.

The talk covered a lot of things that are happening with SIG Release. The team is still moving from the original Google infrastructure to the new infra for the project. It's great that Google donated so much, but the project doesn't want to be too dependent on one company. The team has also been working with SIG Docs on a release checklist.

If you want to get involved with SIG Release, they have a program where people can shadow the current members to learn. You can find more info here. From what I've heard, it's a great program.

Conclusion

That was it for my KubeCon Chicago. Overall, I had a great time and am glad I could attend. I would have loved to see more of Chicago (the only pizza I had was Detroit-style), but hopefully I can make it back there.

I think I enjoyed the April event in Amsterdam a bit more. It feels to me like the US events are not bouncing back from the pandemic as well as the European ones. San Diego, the last KubeCon before the pandemic, had something like 15,000 attendees. I heard that Chicago was more like 8,000 to 9,000 registered, but the crowd didn’t feel that big.

There are additional reasons for this besides COVID-19, like companies cutting travel budgets. I know some people who had to travel to Detroit last year on their own dimes, and that may still have been the case.

But even a KubeCon that's a bit smaller is a fantastic time for me. I got to see so many friends from the community and learn some things, too.

I don't know if I'll be in Paris next spring. It will depend a lot on where I'm working. But I should be able to attend the next North American event in Salt Lake City.