In this episode of CloudSkills.fm I catch up with Kelsey Hightower about how to navigate your career in tech and what the Kubernetes landscape is going to look like in 2020.

You can listen to this episode here:
https://cloudskills.fm/056

Kelsey Hightower is a Staff Developer Advocate at Google Cloud Platform and has worn every hat possible throughout his career in tech but most enjoys leadership roles focused on making things happen and shipping software. Kelsey is a strong open source advocate focused on building simple tools that make people smile. When he is not slinging Go code, you can catch him giving technical workshops covering everything from programming and system administration, to his favorite Linux distro of the month.

• Kelsey on Twitter: @kelseyhightower
• Kelsey’s KubeCon 2019 Keynote
• Kubernetes the Hard Way
• Promise Theory—What Is It?
• Certified Kubernetes Administrator (CKA) Program

Full Transcript:

Mike Pfeiffer:
So, Kelsey, I really appreciate you taking the time out to be here on the show with us today. How are you doing?

Kelsey Hightower:
I’m doing fantastic, and I’m really looking forward to this one.

Mike Pfeiffer:
Yeah, me too. We’ve been talking about Kubernetes all year on this podcast. Lots of people are getting kind of pulled into it. I know that you’re kind of the main champion at Google for educating everybody about what’s happening with all of this new stuff. So maybe we could start with your current impression of things as they are today, what you’re seeing in the field. Lots of people are trying to figure this stuff out. So maybe we could just start there.

Kelsey Hightower:
I think the thing I’m most impressed with is the number of people who are able to have a conversation about distributed systems. Just six years ago, I couldn’t have held a great conversation about distributed systems, just wasn’t my thing. Even as a system administrator managing thousands of machines in some cases, I never thought of it the way we think about these systems today. Seeing the community … and I’m not just talking about experts here, I’m talking about all the people who are new to Kubernetes, learning the terminology, and then having discussions where they take their existing experience, whether that’s with Puppet, Chef, or Ansible, managing VMware, or Microsoft IS. All of those skills showing up now, using the same terminology to describe our own best practices, I think we’re at a really healthy point, which is why I’m now saying, “Hey, what’s next?”

Mike Pfeiffer:
Yeah, I agree. It’s been an interesting year. There’s been so much change. If you even think back three, four years ago, it’s so much different than it used to be. And so, I really love that you said that because I’ve been noticing that, too. And of course, one of the biggest things of doing all this new ways of doing work is being more of a community aspect, really more of working together versus working against each other. That seems to be a key component of the success from what I’m gathering kind of watching what’s going on.

Kelsey Hightower:
Yeah, we’re just at the height of the open-source movement. Because I remember being exposed to open-source in a way where it made technology accessible, there was a small few set of people who have the skills and the ability to implement a lot of these classic systems. We don’t give enough credit to the LS and awk, and all of these commands that are essentially the foundation of this open-source ideas to given us all the power to start leveling up in the game.

Kelsey Hightower:
And now I kind of seeing us doing the same thing but with the community aspect. And reason why the community park gets so much attention now is because we’ve expanded the number of people who are able to contribute in this way. So instead of maybe having 10s or 100s of people, now we’re probably closer to the millions of people. And then when we gather and we come together, that is what we call community.

Mike Pfeiffer:
Yeah, I couldn’t agree more with that. I was actually just having a conversation earlier, before this call today, where someone was talking about how they were able to engage with an engineer at a big cloud company through GitHub, through an issue. That was an open-source project and he was able to basically work with the engineer that supported that feature after a couple of weeks of going back and forth, that kind of access line and that community aspect of the open-source development helped them talk to somebody that actually fixed something in the process.

Mike Pfeiffer:
It’s kind of an interesting time for anybody to really get into it because I believe that he was really getting in there to help with the documentation update, as part of his open-source contribution that started all this. And to your point, it’s really fascinating to see there’s just so many opportunities for people to contribute.

Kelsey Hightower:
You know what’s funny? The first taste, that experience you describe, is definitely magical. I remember getting the first taste of that on [IRIS-C 00:04:36] back in the day.

Mike Pfeiffer:
Right.

Kelsey Hightower:
I was trying to learn Haskell of all programming languages.

Mike Pfeiffer:
Oh, wow. Fun.

Kelsey Hightower:
I was really confused about list comprehensions, I just couldn’t read it in a way that my brain could understand. And I remember jumping into IRIS-C and someone being very patient with me. Breaking down exactly how they worked, why they worked that way, and how they were different from, let’s say what Python does, because it also has list comprehensions. I think what GitHub did was make that experience way more accessible to everyone else.

Mike Pfeiffer:
I agree. I was watching your keynotes from KubeCon not too long ago, and for anybody who hasn’t heard it I’ll link it up in the show notes, you can check it out. It’s the KubeCon 2019 U.S. keynote that Kelsey did. I think probably the biggest takeaway for me listening to that was the points you made about being more inclusive. And I think if you look back 20 years in tech one of the ways that you … sometimes people would build influence by being mean. They would try to be Steve Jobs or try to pull that off, and that’s not really the way forward at this point. It’s not about me kind of leading the charge but it’s more about, how can I rally around or get the people around me to rally around this project, and kind of take it that way.

Mike Pfeiffer:
So do you think that for people listening that they’re kind of trying to figure out where they’re going to fit into this new world, is that something that they should shift their focus to, is more of this community aspect?

Kelsey Hightower:
It’s funny, we talk about community as if it’s a separate thing. You’re human first. You were born into a community first. Engineering is second. The software is second. And when we hole up into a corner and we just try to have a relationship with the computer, that’s weird on its own, but that’s the unnatural bit. The part that is natural is interacting with other humans. And when we can bring some of that natural essence into the work, then I think it becomes more enjoyable.

Kelsey Hightower:
When you talk about forms of leadership, I think there are some cases where people haven’t seen different types of leadership. Some people have kind of seen the alpha male style of leadership or the king at the top of the hill form of leadership, but there’s other forms of leadership. And sometimes you can be a person who just cares about others. You can be a person who really keeps everyone honest and respective of each other. There’s all forms of leadership. I think what I’m hoping we do is that no matter what your background is, your level of expertise … Because remember, whatever your level of expertise is, that’s only for today. Ideally, you’re getting better as you go. So we need everyone as they are to show up and when the time is right, I don’t want to force anyone into anything, but when the time is right show up and be that leader. And sometimes leadership is just as simple as responding to someone’s issue on GitHub with a solution that works for you.

Mike Pfeiffer:
I really agree with that a hundred percent. I think probably for me the biggest hint to a lot of this … because I did that. At the beginning of my career I was very just one-on-one with the computer for a lot. It’s easy to get kind of trapped into that thinking where it’s like it’s just me kind of grinding through doing all of this technical work and then kind of forgetting about the human element that is the default, so after kind of going through that. And then, working at bigger companies like Amazon and Microsoft, and then realizing how much culture actually drives the success of these companies.

Mike Pfeiffer:
When you mentioned leadership, I was thinking about Amazon because they’ve … just because I’ve worked there and I’m so familiar with their leadership principles and all that. There’s so many different ways to lead which is true. But do you think that because of that and the way that the folks like yourself, like Google and Microsoft and Amazon are doing business, that’s going to have to influence customers at some point, that are kind of in that old model where they’re not maybe thinking about open-source. Or maybe they’re not thinking about cross collaboration between teams and things like that. Is that going to be something that the big vendors have a challenge with at getting these other businesses to kind of play up to their level from a cultural perspective do you think?

Kelsey Hightower:
I’m just going to draw on my own experience. I remember, when I first got into big tech, as a consumer, as a customer. I remember, you would just go and buy software from a vendor. And the expectation wasn’t that you would actually fix or contribute to that software. You would just buy it and if you needed some new features you would reach back out to the vendor and you’ll ask, and maybe they deliver it and then you pay for it. So that was the relationship I think in the early beginnings of a lot of this stuff because it was hard and very few people knew how to do it.

Kelsey Hightower:
But if you zoom out over a 20 and 30 year period, a lot of people who work at those vendors, now working at those companies, and they know how to build that software. They know how that software should behave. And when you combine this kind of migration with the open-source nature, these people are starting competing products. Think about all the open-source databases we see. Think about all the open-source programming languages. These things typically were all proprietary. So once you start to have people creating these open-source things, they start to build businesses on top of these open-source projects. And then once you do that, now you’re starting to invite the customer to also be a contributor, and now the coacher around software is shifting because of this open-source nature of building businesses on top. And these are fairly successful business that are driven by community contributions.

Kelsey Hightower:
Now, this isn’t just charity we’re talking about here. This is the ability for a customer with the right skillset to have a say in how the product works, so that way they can get some priority on the features that they need. And then it leads to all these nice things like extensible APIs, the ability to modify the software to fit your own needs without breaking the software. And I think that’s at the core of a lot of the open-source principles.

Mike Pfeiffer:
Yeah, it’s an interesting time definitely in the industry. I’ve noticed the open-source shift really starting to have an impact in the stuff that I’m kind of working on. Especially in this year, but even more so over the last six, seven months. Its been pretty impactful.

Mike Pfeiffer:
But switching gears just a bit. What’s the landscape right now with Kubernetes? There’s lots of progressive companies using it at a great depth, and then there’s a lot of other companies evaluating and trying to see should they even get into that game, and bunch of stuff in between. Your current thoughts on Kubernetes would be kind of interesting for folks listening. I’m sure that there’s a lot of circumstances and different variables, but maybe we could get into that a little bit. What’s 2020 going to look like for Kubernetes?

Kelsey Hightower:
Yeah, if you’re a company, you got 50 VMs and your terraform is right where you want it, your configuration management is right where you want it, things just work. Your team is spending more time on the server ability than they are managing virtual machines. You’re in a sweet spot. Now, while Kubernetes might be very interesting to you, you might not necessarily need it if you have all those other things in play. You might be fine where you are. So you might look up and say, “Wow, this Kubernetes thing looks interesting but we have what we need.” And I would encourage those group of people to probably just pay attention, maybe even kick the tires on it, but I wouldn’t necessarily be rushing to change everything if I have the world I just described.

Kelsey Hightower:
Now, most people that I meet … so I probably go onsite to see between 50 to 100 customers in a year. And what I tend to find at the larger enterprises, they typically feel like they’re made up of 10 different companies. And one company may be pretty far along with some new technology, one company’s area of responsibility is managing all of the systems that currently make money, that’s my substitute for legacy in our structure. In their world, there’s just different needs. So if I was thinking about existing investment, and you find yourself trying to retrofit your infrastructure by bringing in things like service discovery, maybe you want to do automated deployment, maybe you’re thinking about configuration management for the first time, then it gets interesting because at that point Kubernetes represents a way to go from that entire middle journey of going through all of the previous abstractions to zooming to the current state of the world, and maybe you can just make the link to Kubernetes and just package your apps in containers, and leverage Kubernetes going forward.

Kelsey Hightower:
If you already got something like Nomad, or maybe you’re just using something like Docker Swarm, then I think the big question on the table is momentum and community. And what I mean by that, I remember seeing this before when Solaris was the king. I remember learning Solaris before Linux. But then I saw that open-source community come from behind and everyone started to gravitate towards Linyx, the ecosystem did, and then Solaris kind of fell to the wayside. I think we’re seeing Kubernetes come up and challenge some of those existing platforms with the same type of momentum that I saw back then.

Mike Pfeiffer:
Yeah, history does tend to repeat itself a little bit. I’ve been talking about some of the parallels that I’ve seen, since I’ve also been around a long time. That’s really interesting insight. I think there’s a lot to unpack there for a lot of people listening to this, because depending on where you’re starting it’s a different conversation.

Mike Pfeiffer:
And on that question, one of the opinions that is circulating right now, and it makes a lot of sense, is that most companies that do go forward and try to work with Kubernetes are probably going to use it as some type of manage service. Maybe they’re not going to truly build their own Kubernetes cluster because that’s just a lot of work and the skillset required in engineering teams to actually pull that off is substantial. Is that kind of like … do you agree with that notion of people are going to view Kubernetes as a managed service going forward, or is there still merit in people really digging into the depths and knowing how to operate a cluster like an expert would?

Kelsey Hightower:
You know, it’s funny, 15 years ago I got my first Linyx machine. And I open it up and I’m following some tutorials on how to install software, and one of those tutorials is like, “Use VI to edit this file.” I was like, “Okay. VI,” and I gave it the file name. And then from there I made my changes but I couldn’t get out of the editor. I literally didn’t know how to get out of the editor. So I closed the window, I hit the little red button X, and then I noticed my changes were not there. I’m really losing my mind at this point. I do not know how to properly save a file in VI, and I’m looking for now a graphical interface, something I’m used to like Text Pad or something, because I just don’t understand VIM.

Kelsey Hightower:
Now, we could say that VIM is just too complex for anyone to ever use and we should stick to these other things. I think what Kubernetes represents right now is that most people are unfamiliar with Kubernetes, so it’s really hard to judge a system that you have no familiarity with because it’s always hard. Everything is hard if you have no experience with it. Now, once you get a little bit more experience, then you can start to judge and say, “Hey, these areas are probably a little bit more complex than they need to be,” and start to offer some suggestions for improvement. So I think we have to make sure we separate people who are looking at Kubernetes for the very first time and giving them a little bit of time to get their hands dirty, and then come back and say, “What parts of this need improvement?”

Kelsey Hightower:
On the other side of that is we have to make sure that we split the areas of responsibilities around the API boundary. So I do see people today, there’s going to be a lot more people using the Kubernetes API, whether that’s linking it to your CICD system or using [inaudible 00:17:00] to apply directly. That’s one way to go about it. And there’s another group of people whose expertise is rooted in operations, and they know how to manage a cluster, back it up and all of these things. I think that’s just where we are right now, but we just got to be careful about what lens we’re looking at this problem though.

Mike Pfeiffer:
Good point because it does seem like you get into these situations, especially now when things are moving so fast, where there’s a lot of resistance to change. And you’re actually talking to people that are in the business of changing stuff, which is interesting, so I think it’s a good reminder to understand how you’re looking at it, what perspective. Because you’re right, if you are completely brand new, it’s insanely hard, but if you’ve been exposed to it for a year it’s not as hard. So I love that. So it sounds like there’s going to be different opportunities depending on the person, depending on the organization. That’s pretty exciting.

Mike Pfeiffer:
One of the big things that the folks that listen to this show are often working through is certification. They’re ramping up their skills, trying to move their way into cloud computing. I’m curious to have your thoughts on certification for Kubernetes. I know the CNCF Cloud Native Computing Foundation has a couple certifications. One for admins on the OP side, and then one for developers. Do you think that that’s a worthwhile pursuit for folks that are going to be really heavily involved in Kubernetes, maybe in 2020 and moving forward?

Kelsey Hightower:
You know, it’s funny. Someone asked me a question about certifications years ago, and I said something incredibly stupid. I said something like, “Certifications don’t matter. They don’t verify anything. They only verify that you studied a test exam.” The reason why I think that was a very stupid thing to say was because that’s how I learned and got off the ground. I remember, I started with the A plus certification. I was just brand new to all of this. Even how to put together a computer. I started with the A plus certification, which really teaches you about the hardware components and the differences between hardware and software. And then I went to networking plus. I studied for the Cisco stuff. I got Red Hat certified. I was on a roll. And it gave me a way to at least find a north star when I didn’t have one before.

Kelsey Hightower:
So the thing that I love about having the Kubernetes certification … I remember when that whole thing kicked off, they had a group of us kind of evaluate the core competencies to make sure that the test was evaluating the right set of skills so that certification would have meaning. I would go on to find out that people are using one of the guides that I’ve written, Kubernetes The Hard Way, to prepare for that particular certification.

Kelsey Hightower:
So, as a person who’s benefited in the past from having a north start such as a certification, I think it’s a great place to start for most people because you have to be rooted in something. Now once you get that certification, I do believe that your experience will far surpass anything that that certification is going to do for you longterm, I mean 10 years from now. But I think it’s the right place to start for people to get some foundation underneath their feet, to build upon that experience.

Mike Pfeiffer:
I had a similar experience with certs. I started with A plus as well and then kind of ended up doing networking, and kind of went down the Microsoft path after that. But I can also echo, for me, that was key. But even today, even after being in the game for 20 years, I’m still using that tactic to introduce myself to new technologies and to your point, more importantly, go off and actually build stuff later. But I think the opportunity’s really good with this series of certifications on Kubernetes because it’s actually validating that you know what you’re doing based on hands on performance based type of scenario.

Mike Pfeiffer:
So seeing the success in the past of the CCIE on the Cisco side that was all hands on. The Red Hat certified engineer certification was all hands on. I think for me, doing one of these certifications is pretty impactful, like the Kubernetes certifications because you’re actually proving you can do it. You didn’t memorize the answers, you didn’t download anything off the internet. It’s truly you knowing what you’re doing. Seems pretty valuable to me. And a great place to start I think for anybody in their career if they’re brand new to Kubernetes.

Kelsey Hightower:
I also like the accessibility aspect of the exam. Meaning lots of people can’t afford to go to an expensive boot camp, lots of people can’t afford to get a college degree, so having something like this that will help onboard someone to a red hot job market, I can’t do nothing but applaud that whole program.

Mike Pfeiffer:
Yeah, it is pretty awesome. I know that there’s been actually a couple people in our community that have taken the exams and stuff. It’s something that I’m going to definitely do in 2020, at least one of those certs. But it’s funny because every time I look up the study guides they’re like, “You got to read Kelsey’s, Kubernetes The Hard Way.” It’s like the defacto standard. They’re like, “Really take you into the depths.” I’ll also add that to the show notes for anybody that hasn’t seen it yet.

Mike Pfeiffer:
But one of the things that I wanted to touch on real quick was one of the things I picked up off of the keynote and listening to you talk in some other events and things like that, is that not only are you a hard core technologist but you’re a pretty humble guy and that’s kind of a rare combination sometimes. And I think that success for most people would be bigger if they had more humility in a lot of ways that they operate. I know it’s kind of getting outside the technical realm but is that something that you developed over time and kind of … You’re paying attention to what’s happening to that human element. It came up earlier in the conversation. But it seems like you’re very in tune with those types of things. Is that something that took a long time to develop, and is that an important skillset people should be thinking about on top of all this other stuff?

Kelsey Hightower:
Yes. I think we see a lot of examples in the world where maybe being an asshole actually gets you pretty far. Humans tend to have some appreciation for that kind of brass approach to leadership or fame or popularity. I think there’s an aspect of that that people like, and it’s the confidence piece. People embrace people that are confident. Even sometimes when they’re wrong. That confidence is what allows you to make other people feel confident.

Kelsey Hightower:
Me coming up, I decided early on to a couple of things. Number one, to be financially free. And just to find comfort in having control over my own life. So this idea that I would be debt free. Got debt free and stayed that way since my early 20s. And then I started to decide that you know what, these material things don’t define me. Jewelry, the cars, the house, none of that defines me. And I think that took a lot of work over 10, 15 year period to really say, “I am not those things. I am Kelsey, and I have to spend the rest of my life exploring who Kelsey was without those things.” That kept me really grounded.

Kelsey Hightower:
So then I think when a lot of this, what people will consider success, came, I had already found my own identity outside of that, whereas the success didn’t define me. I wasn’t the Puppet Labs guy even though I worked there. People consider me this Kubernetes person, but that’s only a checkpoint in this ever evolving timeline that I have. So I think the part where, you can call it being humble, is because I know there is a much better Kelsey in the future, and I’m willing to embrace the fact that I have so much to learn between now and then.

Kelsey Hightower:
So that kind of keeps me grounded to say, “You know what? The people around me are learning, and so am I.” So that means we’re in the same boat when it comes into the ability to improve. And I think that just becomes who I am. And then the nice thing about that is, I tested this. “Can I just be the humble person? Do I have to be loud? Do I have to be mean to people?” Turns out I don’t, so I don’t go that route. I get to just kind of focus on the parts where if I can share something that I’m learning with other people, that seems to work out well for me. So I’ve embraced that identity which is, I think, my true identity.

Mike Pfeiffer:
Man, that really resonates with me a hundred percent. Man, there’s just so much good stuff there. When I first started out in the industry, I really was stuck on labels. I had an issue of like, “Oh, I’m in this lane and that’s who I am.” The points you made there are super insightful because the reality is you’re continuously evolving a human being and technologist, and I think that that frame of mind puts you and sets you up for success in an environment where you’re doing continuous delivery. You’re changing stuff constantly, you’re always evolving, you’re always trying to get better. So it sounds to me like, not only do we need to figure out how to do that from the tech side, but as individuals we also have to think about that from ourselves. There’s no finish line. There’s always another level to get better, right?

Kelsey Hightower:
Yeah. And that shouldn’t push you to feel like it’s a never ending road. That’s not what we’re saying. We’re not saying you have to go learn all the things. I’m saying be patient and confident in the things that you’re approaching, and as you learn, just know that you’re getting incrementally better than you were the day before. And that’s it. You’re kind of only really racing against yourself in this regard. And that means that you can ignore a few things that are around you that just don’t make sense for you right now, and I think that’s kind of at the heart of it. But the things that you do find interest in, things that are within your realm of responsibility, that’s where I like to double down and make sure that I can go as deep as possible. Sometimes that’s, in this case, starting with certifications, or just talking to other people and asking questions. And every question that you ask is another opportunity for you to level up.

Mike Pfeiffer:
So true. Yeah, always kind of looking for … Yeah, I think for me, kind of the comment before … when I think about it as kind of a never ending thing, I’m not talking about, “Oh, you got to grind it on, it’s going to be painful and you got to rub that all the time.” But I think to me, if I think about it’s kind of like what you said, was knowing that I can always come in and contribute in a different way. I can always grow more going forward. I think that’s an important message. Because I think when you have that perspective and realize that you can always have an opportunity for that, then maybe you can start to become a little bit more open to some of these changes.

Mike Pfeiffer:
But maybe switching gear just a little bit to some more technical stuff. You talk to so many people, right? Obviously you’re visiting customers and doing all these shows. Is there a common pattern of a barrier to entry for Kubernetes that you’ve seen out there, or any major stumbling blocks that people listening might be able to kind of get around based off of what you’ve seen out there?

Kelsey Hightower:
I think for the sake of this show I’ll talk about some of the ones specific to Kubernetes that I see, but the actual things and patterns are roughly the same for any other thing. Kubernetes itself, from the outside, promises a lot. It promises to make you cloud native. For some people it promises to make them taller. So people have an unbound expectation of what Kubernetes is going to actually do for them. That’s okay because sometimes that serves as inspiration and motivation, to really kick the tires and plow through some of the initial learning curve.

Kelsey Hightower:
So one of the challenges is going to be, where does Kubernetes sit in your current infrastructure? If you’ve already got VMs, maybe your company’s been around for a little while so there’s mainframes, there’s VMs. You’ve bought everything every vendor has been trying to sell you. Now your sales rep switches companies. Now they’re selling you Kubernetes. So what do you do? Do you go off and replace everything with Kubernetes? And that creates so much tension within an enterprise because a lot of times when you have that initial conversation, “Hey, here’s this new thing.” In many ways you’re almost saying that your thing that you’re currently doing is old and outdated, and we should replace it. That’s not … people should mean when they approach this problem.

Kelsey Hightower:
So I think what people have to do is sit down and say, “What do we currently have?” This exercise of, what do we currently have, is the number one thing I see missing from the Kubernetes conversation. Since people don’t know what they have, they just get into these arguments. So they’ll go and say, “Hey, we want Kubernetes because they have service discovery, declarative configuration, and X, Y, Z.” And then they go talk to someone else who says, “Well, we already have service discovery, we’re using Consul. We already have declarative configuration, we’re using Ansible,” since you don’t know what you have you approach it in the wrong way.

Kelsey Hightower:
I think the different approach is, you want to show what Kubernetes can do for you that you can’t do today. Or, how you can start to pay down maybe some tech debt, maybe you can get rid of a custom stat in favor of one that’s becoming an industry standard. I think those are the approaches that I kind of see people miss the boat on in terms of adopting Kubernetes.

Mike Pfeiffer:
Read the tweets a couple days ago, I think, or a week ago or something, that you had put out there and said, “A future where Kubernetes Clusters disappear and we just leverage Kubernetes style APIs that deploy workloads is fast approaching.” I wanted to get your thoughts on that a little bit more in depth, but also, are you thinking there’s a huge opportunity for somebody to come in and, to your point earlier, really build on top of this thing where people are talking about multi-cloud and stuff with Kubernetes. Is that kind of where this is going, we’re going to see vendors now building management tools that span these different cloud platforms and on-premises environments to manage Kubernetes?

Kelsey Hightower:
To me this is probably the most interesting thing to pay attention to right now, in 2019 to 2020 going forward with Kubernetes. I remember … the web, right? You build a web page, you use a web server to host that web page, and use a web browser to view that web page. That trifecta. Some of us call that the internet. We don’t really think about all the layers in between. One thing to come out of that trifecta was the concept of RESTful APIs. This idea that we could take, get, put, post, delete, those bodies, those headers, and start to build something that’s not quite a webpage. We’re building these APIs that you can call to automate things and give protocol and meaning, to intentions that people want to do, and then you end up with all of these services like PayPal or Twitch. All of these APIs that allow us to do all this stuff that doesn’t necessarily have a webpage, so we’re able to decouple those two.

Kelsey Hightower:
Kubernetes, people are starting to discover the power of the Kubernetes API, and it’s often described as custom resource definitions or operators. And the way this works is, when you think about the intention, I want a database. I want a database in this region of this world with this much memory. That’s a very declarative thing to say. And ideally, you can give that definition to some tool, and that tool will bring it to life.

Kelsey Hightower:
The first tool that we built like this in the world of Kubernetes is a container orchestration system. This is the thing that people are using today, so when you think Kubernetes, you think container orchestration platform. But when we extract out that API, that control plane component, now you start to think of Kubernetes style APIs. This is a term I like to use, infrastructure as data. This is slightly different than infrastructure as code in meaning that, instead of having people write code to describe what they want, and that’s a valid approach, instead, we move all the logic into these control loops, often called operators, and those control loops expose just enough configuration to contract between the person who needs some infrastructure and the controller that knows how to make it happen and keep it that way. I think that part is starting to become a building block for new systems whether you plan to run containers or not.

Mike Pfeiffer:
Yeah, that aspect of keeping stuff in its desired state is pretty magical when you see it in Kubernetes. Even in things, like if you’re still doing old school VMs and you’re doing auto scaling groups and some of these other cloud platforms out there that kind of has that element to it but it’s not as fast, it’s not as easy to build out. And I couldn’t agree more with you. I think that that aspect of all of this technology is incredibly fascinating.

Mike Pfeiffer:
I wanted to also ask you, because you see people doing Kubernetes on all these … they’re putting it in all these weird places, like raspberry pies and all this different stuff. Is there any weird use cases you’ve seen Kubernetes used for that was totally just shock you?

Kelsey Hightower:
Oh, nothing shocks me at this point because I just think Kubernetes as a control plane API is just going to find its way in more places. I think I’m probably shocked that people haven’t picked that up as fast, so we kind of see it on the smaller systems. We kind of see it anywhere people think about deploying applications. I think we’re going to start to see is it find its way in just other systems that have nothing to do with deploying an application. And I think that’s the thing that’s going to surprise me when I see that for the first time.

Kelsey Hightower:
I want to go back to one of the previous topics just for a second. This idea of declared, desired state and having it converge. It’s not a new one. I was first exposed to it via Promise Theory, so if you don’t know who Mark Burgess is you should go check him out. He kind of wrote a book around Premise Theory. Promise Theory could be said that it’s the foundation to configuration management. So Puppet, Chef, CFEngine, all of these things were attempting to implement Promise Theory for, I don’t know, deployments in managing machine. So you write your Puppet module or your Chef cookbook, you’re describing your desired state, and ideally you can give it to the system and it will try to converge on that promise using the things that were built into it.

Kelsey Hightower:
The problem with that though is with the infrastructure as code, that movement, was that the logic kind of lived on both sides. Some of the logic lived in the description language itself, so it was less declarative than it could have been, and in some of the logic layers inside of the agents that tried to resolve to the desired state. What Kubernetes does is doubles down on this Promise Theory by saying, “We’re going to push all of the logic into the control loop, and only the configuration gets exposed to the outside.” So you can’t put four loops and IF statements in the Kubernetes configuration style APIs, you have to put all of that in the control loop. Then it really enforces this idea that what you declare is what you get.

Mike Pfeiffer:
Yeah, it’s powerful stuff, man. It’s really, really cool. Very interesting. The other question I wanted to kind of get into as well is, this could seem pretty intimidating, somebody brand new, kind of talked about that. A lot of the people I work with are coming from a Microsoft enterprise background, so they’ve been working on Windows servers for a long time. They, sometimes, haven’t even started looking in containers just yet, and so obviously it’s harder to understand really the essence of Kubernetes before you understand the basics of containers in a lot of cases.

Mike Pfeiffer:
I’m wondering if you’ve dealt with this in the past, or things like that, where somebody from that type of a background, if they’re looking at this new world, is there a couple of places that they could latch onto. Obviously you publish lots of content out there to help people get started. But I’m kind of curious, where could somebody completely brand new, they’re in IT but they’re just completely brand new to all this in a major way, what’s the best place for them to start?

Kelsey Hightower:
I think the best place to start is try to really figure out what you already know. The reason why I say this is, I’ve actually worked in a lot of Windows environments, and there are some people who really mastered the GUI tools. You open up the service window, you click on this, you click on that, you hit save, it’s working. You move onto the next server. You open this, you open that, you click on that and you save. And then, maybe you write a script to do that.

Kelsey Hightower:
And a lot of times what happens is, we may not have taken the time to learn the fundamentals of processes. A process will start, buy into some resources, the file systems, the UID, a port, and this is the same for Windows. Windows has services and processes, just like Unix does. But a lot of people have never really thought about that fundamental component to running an application. They just say, “Oh, no, I used this control center thing and it runs the application for me, just put the EXE file here.”

Kelsey Hightower:
So since they don’t have that connection, when they look at Kubernetes they think, “Oh, this is a whole new way of thinking about things.” It’s actually not. And if I took a Windows application, lets say it’s written in .NET, I may decide to build a .EXE, and in that EXE it’s going to have most of the things that the application needs, and I’m going to copy that to a server and execute it, or maybe in an MSI to have an installer on top. So maybe the EXE is a little too low level, maybe I need to prep the system beforehand. So maybe I can wrap that EXE in a MSI and it gives me this installation tool so I can actually keep track of what I installed, and I can also have a foundation for removing it and its dependencies. This is great. So raw source code, executables, and then these packaging formats.

Kelsey Hightower:
And then those can be distributed across lots of Windows servers using some of the Windows management tools. And then we start treating multiple Windows machines like a cluster. Think about it. If you manage a D active directory for example, you may decide to put multiple Windows servers in the same domain. At that point, you’re treating them like a collection of machines, and then using a D, you can have various policies about who’s allowed to log into that system, what’s allowed to run on those systems, and then the control plane can then convert that to all the other systems. If you understood those fundamentals strictly from the Windows world, you have about 85% of what you need to understand how Kubernetes works.

Mike Pfeiffer:
That’s really powerful, that analogy. I love it because I think you’re spot on, but it’s cool to hear 80%. Essentially that concept is the main thing, and then beyond that it’s just the mechanics of getting Kubernetes to do its thing.

Kelsey Hightower:
Yeah, so instead of EXEs or MSIs, we’re building Docker images. And instead of, maybe domains, we’re building clusters. Instead of IS, we’re using things like Envoy, or some other low bouncer technology. So a lot of times you’re just really mapping some of the things you know [inaudible 00:40:03], they just want to package apps a different way. Well, how do they distribute the apps? “Oh, you have to write a config like this and you submit that to the control plane.” “Okay, that makes sense to me.” So it’s that mental mapping. It’s that last mile, that last 15%, that really connects all the dots.

Mike Pfeiffer:
Yeah, that’s awesome stuff, man. I really love it. So, Kelsey, I really enjoyed talking to you on this episode today. In the show notes we’re going to add a bunch of stuff. We’ll put a lot of the things that we talked about, the link to the GitHub repository, where Kubernetes The Hard Way is published, and we’ll put the KubeCon conference keynote that Kelsey did out there. Kelsey, anything else that we should add to the show notes or any other big takeaways for folks that they should look at coming out of this episode?

Kelsey Hightower:
No. I think, honestly, no matter where you’re coming from, you have a place in the Kubernetes community if you want it. And don’t sell yourself short. A lot of the things you’re working on right now are the foundational skillsets to adopt anything that’s considered new now or in the future.

Mike Pfeiffer:
Awesome. Well, Kelsey Hightowerightower, I really appreciate you taking the time out to come on the show. Thanks so much. We’ll see you, hopefully, maybe next year on another episode.

Kelsey Hightower:
Awesome.