In this episode we catch up with Abel Wang from Microsoft to talk about all things Azure DevOps and how people are using these services in the real-world.

You can listen to the full episode here:
https://cloudskills.fm/057

Here are the resources we discussed on the show.

• Abel on Twitter: @AbelSquidHead
• What is Azure DevOps?
• Abel’s Blog series on Real World DevOps
• Microsoft’s Official Azure DevOps Blog

Abel Wang is a Principal Cloud Advocate and DevOps Lead at Microsoft with a background in Azure and application development.

Full Transcript:

Mike Pfeiffer:
All right everybody, welcome back to another episode of CloudSkills FM. Super excited to have you today. And my guest on this episode is Abel Wang from Microsoft. Abel, how’s it going?

Abel Wang:
Very good. So happy to be here.

Mike Pfeiffer:
Yeah, I’m super excited to have you here as well. For everybody that’s listening. I know that we’ve got a lot of listeners that understand what’s going on with Microsoft and DevOps probably know you, but for anybody else who doesn’t know you, maybe you could introduce yourself, talk about your background a little bits, all that kind of stuff.

Abel Wang:
Sure. So my name is Abel. I’m a principal cloud advocate at Microsoft and also the DevOps lead. So I work very closely with the Azure DevOps team at Microsoft, the product group. And I fly around the world talking about DevOps and how we can make everything better.

Mike Pfeiffer:
I see you guys out there a lot. It’s like you guys are traveling all over the place. How many countries have you been to like last year?

Abel Wang:
It might be easier to see which companies I have not gotten to. Right. Well, funny enough, last year I got diagnosed with cancer, so I like traveled less. But even then if I look back, I still travel way more than a normal human really should. But hopefully we’ll be doing less of that now. But for a while there, once we started this advocacy group, it was really like, okay, you got to meet the developers where they are and be able to communicate with them.

So I didn’t know what that meant. So the first thing I thought was you know what, if I try to hit every single conference I can get to, then I can get face time with developers, figure out what works for them, what doesn’t work for them. Right. So that’s one of the reasons why I started flying everywhere.

Mike Pfeiffer:
Yeah. And I’m glad you’re doing better now. That’s awesome that you’re back in it and super grateful to have you taking the time today. I know it’s a busy time for you guys, whether you’re traveling around or even when you’re just in the office. So what’s a typical work week look like for you? Are you helping customers? Are you working with people on the DevOps team at Microsoft or?

Abel Wang:
Pretty much everything. So it changes from week to week, but it can go anywhere from if I’m not traveling, then I’m trying to write all sides of content or creating video types of content so people can be educated and see what Microsoft has to offer, right. In terms of the DevOps space. And so a lot of what I do is creating content. Of course, I get constantly pinged by sales teams as well saying, can you speak to this customer? And I’m kind of technically I’m not a salesperson. And they’re like, yeah, but they really want to talk to you. And I’m like, okay, fine. So I did spend a little bit of time, like a year doing sales at Microsoft.

So I feel very, I have strong feelings about sales and I really feel for the sales people because of their quota. I remember how horrible quotas were so very sympathetic. So I’m always like, okay, fine. You convince me, I’ll go help you off.

Mike Pfeiffer:
Nice, Yeah, I could see how that would be happening to you. But since you’re so visible and you’re out there all the time, I’m sure everybody wants a piece of your time. So it’s even more like speaks to my gratitude for you being on the show. But let me switch gears into Azure DevOps. I know that’s, it’s pretty exciting. I got to say the deeper I get into it, the more I’m realizing that like this is the glue man. Getting past the hello world stuff and just understanding the components now or like making this sucker dance with Azure DevOps. Do you think that this is something that’s going to be popping up on more people’s radar going forward in 2020?

Abel Wang:
I hope so. I mean Azure DevOps is one of those products that I have been associated with since the beginning. Right? So it all started off with TFS 2005 I think. And it took us so long to write TFS 2005 but didn’t come out until 2006. So I’ve been involved with TFS since day one and I’m very attached to it. But what I find is fascinating even back in the TFS 2005 days, I mean, there was nothing really, not a whole lot of other stuff out there in that type of space. And it was okay, but where things started getting really exciting, I think is when Satya Nadella came aboard and one of the first things he did was he asked Microsoft, well, what are you guys using for your DevOps tools? Are you using your own Azure DevOps to do your DevOps?

And we were all kind of like, well, sort of, kind of, nah, okay, fine. No, we’re not. So he was just like, okay, that’s ridiculous. You all have to use your own products to know what’s good and what’s bad, right? So then you start looking at let’s say between 2010, 2015, right? Closer to 2015 timeframe. All of a sudden, as your DevOps couple things start happening. Azure DevOps moved into the cloud and I think we called it Visual Studio Online at that point. We have trouble naming things. But anyway, all of a sudden you started to seeing rapid improvements with Azure DevOps and why is because we were using Azure DevOps ourselves, right? And all of a sudden the warts, it’s very different when somebody tells you, okay, there’s a wart over here.

Whereas if you actually use the product and you see that wart every single day, and since we own the source code, we’re kind of like, hey, we better fix that because this sucks. Right? Everybody started seeing this rapid improvement in Azure DevOps until today I think it’s one of the best DevOps products out there. And I don’t say that just because I’m a Microsoft person. Interesting enough, I’m not all about Azure DevOps. My job is to help everybody get into Azure using DevOps best practices, no matter what type of tools they do. So I just did a whole video series of, okay, using GitHub, how do I go into Azure using GitHub? Or what if I’m using GitHub, how do I get into Azure using GitHub?

Or what if I’m using Jenkins and Spinnaker or whatever? Right? So the tooling really doesn’t matter and I use all the different toolings because I’m able to help our customers. However, Azure DevOps is a really, really cool product. It works well.

Mike Pfeiffer:
Yeah, I have to echo that because I’ve worked with lots of different tools and services and I’m continuously blown away by the depth at which you can get into with Azure DevOps. And I think that it speaks to what you said, that it was really born out of a product that’s been around for over 15 years, right? Almost 20 at this point. So I think a lot of that lessons learned has been rolled into the product as well along on the way. And to your point now you guys are dogfooding so you’re using it internally in and even if you’re watching the blog and stuff, it’s like you guys are talking about what you’re doing and each three weeks sprint, right?

Abel Wang:
Oh yeah. So we try to be extremely transparent on what we’re doing and what we’re going to be doing in the future. Right? So if you go to the website there literally is a timeline that shows what we’re going to be doing. So a lot of times I get people asking me, okay, well what are you guys going to be working on for the next year? I just point them to the website and say you can take a look and check often because that list will change, right? Because we listen to our customers and depending on what our customers are telling us, that can change the priority of our backlog. But our backlog is open for everybody to see.

Mike Pfeiffer:
The transparency is really big. I was talking to somebody a couple of weeks ago and I actually mentioned this on another show recently. I was just talking to a guy and he was talking about how he was engaging with somebody at Microsoft through an open source Azure DevOps repository. He was initially just doing a documentation update and he got into conversations with the engineer and next thing he know the engineer responsible supporting that feature or fix something for this customer. He’s like literally talking to an engineer at Microsoft. That’s like a totally new thing for you guys. Right? For the most part.

Abel Wang:
Not a new thing. I’m glad you brought it up because I’ve literally watched this transition that happened right where 10 years ago, good luck getting a hold of any engineer that’s working on any product at Microsoft. You have a problem with any of our products who do you contact? There’s no point of contact. Right? And now, especially with the Azure DevOps team, we kind of reorganized everything so that instead of writing everything in layers, like the data layer and the UI layer and the backend layer, what we ended up doing is we split up our teams so that each team owns a feature from beginning to end, right? So we split it up like vertically. So you own everything, you own the UI, you own the database layer, you own the back end code, even on your own pipelines, right?

You get your chunk of code into production. But because everything is vertical like this, where each team owns an entire feature, it becomes incredibly easy for our customers to be like, let’s say you’re using Azure boards and you run into a problem, you can complain about it on Twitter. We love doing that and someone like me will see it and I can immediately point that to the program manager for that specific feature like or the program manager for that specific feature might see that tweet and immediately respond, right?

So this is really exciting for Microsoft because for the first time in a really long time, we are able to communicate with our customers directly now where they can reach out to us directly. And when we tried to do this through a variety of mediums, whether it’s through GitHub issues or whether it’s through Twitter or any of the other social media platforms or anything. So it’s so much easier for customers to reach us than never before.

Mike Pfeiffer:
Yeah, it’s amazing. And I think in addition to the GitHub stuff that I was having that conversation with the customer about what you mentioned is the Twitter and the social aspect. You guys actually have a hashtag here your point, right? Like if I wanted to get ahold of you guys, I could do a hashtag and say, hey, this isn’t working right.

Abel Wang:
Yeah, absolutely. So the team that I’m on, we have one of the most ridiculous names in tech, I think. We ended up calling ourselves the league of extraordinary cloud DevOps advocates. And this was totally a joke that we were texting back and forth each other when we were just forming. We thought it was hilarious because it kind of is. And then somebody tweeted it and then it became a thing and then it’s kind of like, Oh, now we’re stuck with it. This is weird. But the fun part is if people have any type of either questions on DevOps into Azure or the product Azure DevOps services specifically, if they tweet out us and hashtag L-O-E-C-D-A, that literally lights up a team room for us. We see that and we’ll answer your questions. And here’s the thing, clearly I can’t answer every question in the world, but if I get a question that I don’t know the answer to, I will find the people at Microsoft that can answer it. So I will get you an answer one way or the other.

Oftentimes like people will tweet at me hashtag L-O-E-C-D-A and if my teammates have haven’t jumped on it fast enough, I can look at it and be like, yeah, I have no idea. But I do know the PM involved and I’ll point the PM to it and then they can immediately start answering their questions or gathering feedback or doing whatever.

Mike Pfeiffer:
Yeah. It really speaks to the vibrancy of the Microsoft community. It’s always been really good, but now it’s gotten so much better and things like this really amplify it a lot. But since you’re doing so many demos all the time, I would love to ask you like what’s your favorite, I mean it’s probably hard to pick one. There’s so many cool ones. I bet you’re geeking out with. But do you have any ones that might like be really cool to mention that people might not even realize they could do Azure DevOps?

Abel Wang:
I think one of my favorite things, so it does Azure DevOps for those that don’t know, it’s pretty neat. You can use it to do pretty much everything you need in this awkward project, right? So it consists of five separate services that are built directly into Azure and using these five different services, you’re pretty much able to do whatever you need to in a software project. So for instance, there’s a work item tracking section where you can track any unit of work, like your user stories and your bugs and your impediments and things like that. And there’s a full set of visual tools that you can use. There’s also a source control system service in there. There’s also a CICB system that’s built into it. There’s also a repository and there’s also a manual test case management system as well. Right. So pretty much everything you need for your software.

What I really like about it is part of the CICD system. Yes, there’s a fully customized CICD platform where you can call it Azure pipelines where you can do literally anything that you want to build and deploy your application for any language pushing it into any platform. I think to Azure, of course you can’t. I push and I build and deploy a Java app into Linux. Absolutely you can. Linux server that’s hosted on AWS. Actually yes you can, again, I always say this because it’s true. That would make me cry because I want you to Azure, but our tooling literally doesn’t care. So two things. I love the fact that Azure DevOps is for any language targeting any platform, right? And I think a lot of people don’t realize that yet, even though I say that every chance I get because they think, oh, Azure DevOps, it’s only for Azure.

It’s only for Microsoft tech. I can use it for any language targeting any platform. More specifically part of it that I really love, there is a feature inside of pipelines that I really like, which is the automated approval gates, right? So there has always been the ability to have manual approvers when you go from one stage to the next stage, the next stage. So let’s say I’m going from dev to QA. When you leave a stage, you can assign a group of manual approvers, and this can be a list of person or list of people where every person on that list has to degree before it can pass through that gate. Or it can be a group of people if one person who the group approval folks through the gate. Or it can be a combination of listening groups, right? So this is cool.

You kind of need that to say, okay, I’ll go from dev to QA, I need the approval of the QA manager, whoever that and get that goes on through it, right? So that part, a lot of people have. We also have in there the ability to have gates that are determined using code, right? So in an automated fashion, this gate can determine whether to turn green or to turn red. To pass it on through the next stage or not. And out of the box, you get a whole bunch of these automated gates that are already built, right?

So there is an automated gate, for instance that uses, if you happen to use Azure boards, right? And track your bugs through Azure boards and stuff, you can create a gate that’s based off of a workout inquiry, right? So for instance internally, we use Azure DevOps to deploy Azure DevOps into the public.

So we have an automated gate that will literally sit there and look for blocking bucks. So if anybody puts in a blocking bug for this specific project, it will automatically stop that release and bam, that’s it, right? So we need to freeze our release. Super easy for us to do. We just create a blocking bug for that. We’re done. So that’s a gate that’s built into it as well. Another gate that’s built into it, you are able to use a gate that’s like linked into Azure like application insight right together telemetry. So you can say stuff like, okay, if I push into, let’s say my beta environment, I’d run a bunch of automated UI tests. If I get too many exceptions or if performance is too bad, guess what? Just automatically block that gate, right? So we don’t even need human intervention where it starts getting cool though.

So you get like four or five different automated gates out of the box. You can write your own custom gates too, so you can point a gate to a rest API or an Azure function, right? So at that point in time, if it’s code, you can make these automated gates do anything right? Including things like, okay, I can harness the power of AI. All right, so this is kind of cool. One of the things that we did in Azure pipelines as well, and when we deployed Azure pipelines is we monitor Twitter. So the way that we deploy is we deployed through rings, because Azure DevOps services services the entire globe and it can’t go down, right? Because these are people source code, they’re building release pipelines, their product plans, things like that, that goes down. Other companies can’t work, which is you can’t have that.

So it has to be up 24/7. So when we roll out a change, we want to roll it out very, very slowly and carefully. So we roll it out into rings, right? Ring zero is going to be just us and engineering where we use Azure DevOps for like two days or so. And during that time we’re gathering telemetry, we’re gathering feedback, and if it looks good, it goes onto the next ring. If it doesn’t, it can automatically get blocked, right? If telemetry tells us performance is bad or acceptance or happening can automatically block that. Ring one after it passes through us, it goes to ring one, which is I think it’s Brazil. Yay. Lucky Brazil, right? They get it. They’re the freshest. The first outside customers that see our new code. The reason why we picked Brazil is because they were the smallest scale unit.

So that would affect people the least. But back to the gates, one of the things that we did is we use this automated gate to look at Twitter sentiment. So if our MVPs from Brazil all started tweeting things like, Oh my goodness, this sucks, blah, blah, blah. Latest release is causing all sorts of bugs, we can gather Twitter sentiment and automatically block that release right then in there as well.

Mike Pfeiffer:
Yeah. So the release gates are insanely cool. There’s so much you can do in there and it seems kind of like you said you can write one for yourself and that really makes it to where now this is a blank canvas because you could write your own functions if you want to. And it was really easy to do that.

Abel Wang:
Exactly. Your imagination becomes the limit now. So a lot of people use things like service now, right?

And you’d have the service now ticket approved or whatever before you can go in from one environment to the next. So people have built automated gates that do that type of stuff. Right. I was recently working in the hospital or I visited a hospital where they literally could not push into production until there was a physical document that was signed and uploaded to their DocuSign server. Right. That’s their process and their rules involved. And so this became a huge bottleneck because now you have to physically contact someone and be like, okay, you need to sign this document and upload it to the DocuSign. Now somebody has to keep on checking that and when it’s finally sighed, then you can push to production, right? So what we did was I heard that and I was like, why don’t you just rate an automated gate?

Because DocuSign has a full set of rest API so you can do stuff better. So just keep on painting and say, hey document sign. If it has been signed, hurray for that, passed on through. If it hasn’t been signed, wait five minutes and then try again. Right? And if it hasn’t been designed in like a week, then go ahead and time out because clearly something’s not working right?

So you can set the polling frequency and you can also set the time out times as well for these gates. So again, your imagination is the limit. What can you do with these gates? All kinds of stuff. So another instance where somebody did a cool thing with the gate I was working with, see what company was this? What I’m trying to say is your imagination is the limit with these gates. There are all sorts of stuff that we do that we need a yes or no answer to before we can push to the next stage.

And now you can pretty much automate just about any of it. In a recent demo that I did, I showed another way that I can use the gate. For instance, there is a certain thing inside of Azure that I wanted to provision during my pipeline, right? Using infrastructure as code. So what I wanted to provision was a front door instance. And so for those that don’t know, a front door is a Uber load balancer inside of Azure that can load balance from around the world. And there’s all sorts of millions different features inside of it.

But I just wanted one of the features, which is it lets you create a table where you can basically define all your backend URLs and have it masked so that looks like it’s coming from the front, right? So you can have a whole bunch of these backend URLs, but then it all looks like it’s coming from one outside URL, which is great because with a microservices architecture that we love to do these days, all of a sudden my backend trying to reach the right service becomes a pain because then you start doing the cross domain stuff.

You get cores issues, but if we all go through front door, it looks like it’s coming through one URL. It solve that problem for me, I was like, all right, cool. I’ll use front door. Front door unfortunately I can’t, is set up in a weird way where I can’t provision it inside of Azure until DNF has propagated from CloudFlare, which is where I’m starting my DNS value all the way down to the local machine. Why? I don’t know the details. All I know is that you have to wait until DNS is propagated down to your local machine before you can provision that stock. You can’t set it after the fact.

So I was stuck with the problem of going, okay, within my pipeline I can set my DNS inside of CloudFlare because it has a full set of rest API. But how do I know when that has propagated down locally? Now what do I do? Right? But then I was like, Oh, automated gates, I can create an automated gate that basically starts paying, hey, did the DNS values propagate down locally to what I need it to be? And if it hasn’t, wait five minutes and try again. And when it does go on to the next day or next stage, which is now that I have DNS propagated locally, let’s go ahead and provision at front door for us. So cool stuff. Your imagination is the limit. It really can help you automate that pipeline from beginning to end.

Mike Pfeiffer:
That’s so crazy. I remember like 10 years ago when I first started really diving deep into power show and I was just blown away at the automation that I could do. And now it’s just like 10 acts at this point. But you’re talking a lot about different solutions and stuff. Does Microsoft have anywhere like a repository where they store examples? I mean I’m sharing a doc so you can just go figure out how do I build a custom release gate and all that kind of stuff. But is there like a repo where there’s like all these weird examples that people could use for inspiration or anything like that?

Abel Wang:
There are, and of course GitHub is where we have all of our stuff. However, if I was a dev I wouldn’t start in GitHub, but it’s a great resource and I would go there. If I was a dev starting out, I think the place where I would start would be docs.microsoft.com. I think if you just go to docs.com point you to docs.microsoft.com.

We’ve been working extremely hard in creating better docs, right? Good docs equals developers can now go in and actually figure things out and do things. Whereas instead of needing all this tribal knowledge that you only gained from using the same product for 10 years in a row, right? And so we’ve been working extremely hard on making good docs pages. It can get better still, but it’s so much better than it was even bikers even three years ago. Right?

So we have docs on everything inside of Microsoft including stuff with Azure DevOps. So, and of course it breaks it down into all sorts of stuff. There are walkthroughs that will walk you through what you can do and there are also links that point you to repost that actually show you more in depth what you need to do as well.

Mike Pfeiffer:
The documentation is awesome. People always ask me, what’s the best book on this new Azure thing or whatever. And I always tell them to go to the Docs because there’s literally armies of people writing documentation and you look at the date and be like, oh, it was just updated yesterday.

Abel Wang:
Armies of full time employees. Right. And one of the coolest things is that we’ve opened source our docs as well. So customers to look at our docs, find an error because errors happen and they’re fascinating to realize that you know what, they can just create a pull request and fix it just like that.

So that’s a great thing too because I remember a timeframe when Microsoft was not about open source and fast forward to 2019, 2020 we are all about open source now, right? So that’s been a great transition as well.

Mike Pfeiffer:
That’s been a huge one and it’s actually gotten to the point now where it’s obvious from the outside. I think you guys have been on that journey for a while, a couple of years, right? A few years.

Abel Wang:
Oh yeah, for a long haul.

Mike Pfeiffer:
But it’s really becoming obvious if you’re watching from the outside that there’s something going on differently at Microsoft, it’s very clear. You guys are moving faster than ever. But there’s also a big focus on more of the human elements and being more considered about everybody and inclusivity, all that kind of stuff. These are all culture of Microsoft and all these other bigger companies too that are going more progressive and moving faster. Do you think that’s going to challenge the customers to play up to that level and is that going to be a struggle for companies to be able to kind of do what you guys are doing which is not very common actually.

Abel Wang:
I think industry in general is moving towards that direction and the tech field is always fun to look at because we moved so quickly. So I think in a lot of sense we’re on the bleeding edge of, but on top of that we’re on the bleeding edge of a lot of things because we move so quickly because just the nature of our industry itself. But I think it’s kind of a good, that’s like the Canary in the coal mine thing. What happens to tech slowly filters down into all the other industries as well. Even the slower moving industry is like, the finance world or banking or healthcare or insurance or things that traditionally move at a very, very slow and deliberate type of pace. These types of changes float down as well. Yeah. I’m sorry, what was your question again?

Mike Pfeiffer:
I was just wondering if you think customers are going to have a challenge being as efficient as a Microsoft or as an Amazon because I guess what I’m thinking is you guys are using Azure DevOps to build Azure DevOps. That’s going to basically into the product. It’s built for people moving fast, right? And everybody can move fast. I guess what I’m asking is, is Microsoft doing anything or planning on doing anything to help their customers with the nontechnical aspects of those?

Abel Wang:
Yes, absolutely. So a lot of what I do is going around and talking to C level execs and telling them about Microsoft’s transformation journey, right? This is to show them that A, this journey is long and painful and B, but it is vitally important that you make this. And the reason why it’s vitally important is because if you don’t, your competitors will or they have and when they do, they will out innovate you and that will render you absolute.

Mike Pfeiffer:
That’s so true.

Abel Wang:
That’s a very dangerous place to be. And this was, Microsoft made all of these changes. We made drastic, if you look at Microsoft from the 90s early 2000s to the Microsoft of today, drastically different culturally how we deal with our customers, how we do with our competitors, right? I mean just drastically different. We didn’t make those changes just out of the goodness of our hearts. Right?

We saw the writing on the wall and it’s like, look, if we don’t make these changes, we are not going to be out innovated. We were being out innovated by our competitors. We were going to be obsolete. What’s fascinating to me is how far reaching you have to really look to see that because you look at the revenue Microsoft was still bringing in good jillions of dollars. I don’t know the numbers, but just piles and piles of money and projected for the next 10 years, they’re still going to bring in piles and piles of money. Even though Microsoft lost the mobile world where all desktop is less and less important. But these are long tail machines, right? I mean, yes, desktop is less important, but the amount of revenue that just comes in from desktop, licenses, it’s massive and it’s still just going to be there for at least the next five years, 10 years, whatever.

But our leadership is smart enough to look at that and be like, okay, that’s not sustainable short term. Sure. But long term we have to make these changes, right? So and like I said, change is difficult. Change, painful. We’re going to lose people, we’re going to lose productivity, we’re going to lose all sorts of stuff. There has to be a good reason why we made these decisions. These are the reasons why we made it. And when I share these stories with other C level execs that’s the thing that I’m trying to show them, which is, look, I know this is going to be hard, but there’s a reason why we went through these changes because we were going to be obsolete and now that we’ve made these changes, look at how fast this transformation has happened and look how much more productive we can be now.

Mike Pfeiffer:
I mean that’s super important, especially if there’s anybody in any kind of executive listening, because I’m sure you’ve had this issue in the past where you get somebody that wants to buy DevOps and just [inaudible 00:27:25] right?

Abel Wang:
I almost choked on my coffee. I hear that way too much. How do I buy DevOps? You can’t buy DevOps, right? I mean, when we talk about DevOps, right? People always talk with, especially at Microsoft, we talk about the people, the process and the products that you use to enable everything you need to do at DevOps. The tools that you use that almost really doesn’t matter. The process that you need to have, any type of agile process will work fine, right? I mean, in theory, really, you just need a process that, that will let you iterate quickly, yet still deliver code of high enough quality, right?

So there are plenty of processes out there that you can just pick up and start using. The hardest part and the most important part is the people, the culture, right? You have to have a culture that embraces these DevOps concepts. I would argue that if you have the right cultural mindset, it almost, not 100%, but it almost doesn’t matter what type of tooling you’re using. And it almost doesn’t matter what type of process that you’re using. You have to have the right culture. If you have the wrong culture and have the right process and the right tooling, you’re still going to fail, right? That’s how important the people portion to culture. And it’s also the most absolutely hardest thing to change because people are hard, right? People are difficult.

Mike Pfeiffer:
And just like you said, like you got like earn it, right? You have to like take that journey. It’s not going to happen overnight. You can’t buy it off the shelf. But beyond the cultural change that’s incredibly difficult. Is there any sticking points that you see commonly for people? Maybe from a technical perspective for it’s a common stumbling block that we could probably hedge for folks listening?

Abel Wang:
Usually the biggest trouble starts with especially when you’re talking about like the slower moving industries. The problem starts with culture because they are nowhere even close to having this idea of DevOps. I mean they are so far away from DevOps that that thing that I asked them is your entire organization is set up in a waterfall manner, right? Where everything that you do is dependent on you having like a yearlong plan so you can figure out your budget and then here you go, developers start developing.

It’s just everything is baked in that way that clearly things are not working right. The software that comes out is always over budget. Over time they’ll get the bugs, et cetera. So I mean we’ve all been there. So there has to be a better way to do this and you need to make that change first, which is okay. If you can’t even get your software out on time, bug free. And when it comes out, this is what your customers or end users really want or need. That’s the place that needs to be fixed first, right? So it’s like, okay, you need to embrace the agile world first to figure it out, how to build the software that you need first, right? So once they can figure that out again, we need a cultural change and also a change in process, then it comes the fun part of going, okay, fine you know what? You’re actually succeeding in agile now sprint after sprint, you’re getting the software that you said you were going to build and the customers are super happy and everything’s is kind of clicking along.

But where’s the bottleneck now? What’s the pain point? And usually the pain point is great, sprint after sprint, we’re doing this, but we can’t get our code into production fast enough. We have a two week sprint. It takes us a week and a half to get it into production. We’re falling behind and it’s everything’s falling apart. Again, this is just things that happened to teams that makes these transformations all the time. And so then I’m like, okay, well great, now we need to start automating our deployments so that we can actually do all of this on time. Right? Because if we automate it, humans are slow, humans are prone to mistakes, et cetera, et cetera.

So then they start figuring out the automation. And that’s always a little tricky at first. But again, there are a whole bunch of tools that can do this. I love Azure pipelines and Azure DevOps. But again, lots of tools that will allow you to do this. And eventually they get to the point where they can automate their deployments and it’s like, okay, this is cool. But then other things start cropping up, right? Because now you’re like, okay, I can build a software that I said I was going to build in the time that I said I was going to build. I can even deploy it into production fast enough. But how do I now maintain quality in a DevOps world because I have these beautiful pipelines, click a button, it’ll grab the code from source control, push it all the way, even out into production within a couple of minutes.

Am I just pushing bugs into production super, super fast? And that’s when companies now need to look at it and be like, okay, how can we maintain quality even when we’re doing things like pushing code into production every two weeks, every week or whatever. Right. And that’s when they have to start learning that. Sure, the way we used to do quality is we try to bolt it on at the end of our software cycle, right? We read our code and when we were done, we throw it over the fence to the QA people and the QA, they would find my bugs for me and they would find a mountain of bugs that have written and then I’d have to burn through that mountain and then they have to retest, right? They would find my bugs by doing end-to-end functional testing, which is fine.

And that kind of sort of worked. It’s a burning through that mountain of bugs. It’s usually impossible because there’s too many bugs. So you try to get the quality. So it was good enough. But the part that’s scary is that takes forever like end to end functional testing takes weeks, could take months. If your code base is as large as let’s say something like windows, it might even take longer than a year to run full end to end functional testing.

So in the DevOps role, clearly you don’t have the time. So how do you maintain quality? Right? So you’ve got to shift luck. You’ve got to build quality into your code as you’re writing it, right? So as you’re writing it, you need to embrace things like, okay, I need for every line of code that I write, I need to have unit tests that makes sure it’s doing what it’s supposed to be doing.

Which means then you have to learn how to write your code in ways that it’s testable, right? Because if you don’t write your code in a testable manner, get a tests are useless, you can’t test it. Right? So it’s like, okay, you have to change the way that you read your code so that it can be testable. You have to figure out, okay, maybe I can do automated UI test as well. We need to do things like every time we check in code, let’s have a code request to code review. We basically need to shift left. So you build quality into your application as you’re writing it, as opposed to tacking it on the end. And then all of a sudden you realize, oh, we can still move quickly and still maintain quality of the DevOps role. So that’s a big thing that I often talk to with companies making this type of journey.

Of course, after that, and he have other things as well, right? How do I handle databases or security or things like that? And what I always tell people is find what hurts the most for you right now and fix that one thing because you can’t fix everything at once. You don’t even know what hurts everywhere. Find what huts the most, fix one thing at a time, and then for the next sprint, assess, did that help things? If it did hurray for that. Did it help it enough? If it did hurray for that. Find something else. If it didn’t, let’s tweak it. Right? And so you slowly find what hurts the most and then you slowly start adding more and more things until eventually you become a very mature DevOps organization.

And even then, it’s not over. You can’t just buy DevOps. You can’t just magically wave a wand and say, I’m DevOped. DevOps is a journey of continuous iterations and improvement where you look at what you’ve done, see what can be done better for the next iteration that’s improve on that one thing.

Mike Pfeiffer:
I’m fascinated by people where the resistance to change when they’re in the change business. I was having flashbacks when we were talking just now because I mean I would have to agree with what you said. A lot of the hardest parts that I’ve struggled with working with customers has been just the basics. Just like getting version controls set up the right way and just like I had a customer that was already paying the servers and pasting code updates through the RDP session. Like there’s a lot of companies that are getting started for this for the first time and it can be a tough hill to climb.

The other thing that seems to be a theme lately especially, but I know this has been going on for years, is people complaining about the label or the term of DevOps and my stance is quit focusing on that and let’s add value. Like let’s go. You got to, I’m sure you get that all the time, right? And people complaining about the terminology and I would love to hear your perspective on that.

Abel Wang:
So terminology I get that terminology is important and I agree with you right? Because we fight about terminology like crazy and especially at Microsoft, again, naming things is incredibly hard and it’s incredibly difficult. And names have power, which is why it’s important to get it right. So I get that. But the other part of me kind of looks at a lot of these, especially on Twitter, right? A lot of these words or people will start tweeting and just arguing about why is it called this? Or it should be called this or blah blah blah. At some point I fall into the camp of, I don’t care what you call it. You know what, these are concepts that are important. Let’s continuously deliver value, whatever that means, however we need to?

Mike Pfeiffer:
100%.

Abel Wang:
Does it really matter whether it’s called this or whether it’s called that, I just don’t care.

So that’s kind of where I sit. I remember when GitHub started trending and everyone was like, Kelsey Hightower started using GitHub as a phrase. And I remember thinking when I first saw that going, what is GitHub? Like is this something new? Oh my goodness, I’m behind. And I started Googling like crazy, right? Going, Oh, I got to catch up. And I looked at it and I went, Oh wait, this is completely nothing new. This is infrastructure as code checked into source control. Oh wait, this is just DevOps. There’s nothing new here. And some of my colleagues were losing their minds about it and I’ve got to admit, I was one of them. I started losing my mind going this way, that’s too bad, blah, blah, blah. And then I thought about it and I went, who cares?

Right. Is it a good best practice? Actually, yes, it is. Absolutely. So I don’t care what people call it, let’s just use it. Right. And or another term that used to be one of my favorites to combine that with DevSecOps. I was like, why do we have to do dev? Whatever phrase of the moment is ops, right? Just I was like, this is horrible, blah, blah blah. And I was complaining about this and a really good security friend of mine. She looked at me and said, she’s like you know what? I agree with you 100%. This is really still just DevOps practices, but until our customers and the industry starts paying attention to security in their pipelines, yeah, I’m going to still call it DevSecOps. You’re like, once everybody starts using it, then I’ll go back to DevOps.

I was like you know what? They’re enough. So after enough of those conversations, I stopped being such a stickler on racist in terms of just that what’s important? What’s important for me and my customers, for me and the companies that I’m working with. We need to be able to deliver value continuously.

Mike Pfeiffer:
Man I really love that you said.

Abel Wang:
[inaudible 00:38:09].

Mike Pfeiffer:
Yeah. I’m like totally, I mean we’re spot on that together. Like that’s exactly how I think about it. What’s the point of complaining about this? You’re just making an excuse for not building something like let’s go. So that’s my perspective. It’s like we could sit there all day and bicker about the labels, but that’s not adding value to the conversation.

Abel Wang:
Is not adding value. Like I remember I was in a recent conversation where somebody was asking me that the pipelines inside of Azure DevOps, why is it called a release pipeline?

Because you can do so much more than just release your code. Maybe it should be called a deployment pipeline or maybe and other people jumped, oh, but it doesn’t mean it should be called a blah, blah blah pipeline. And I remain mostly silent on that thread because I really didn’t have anything to add because I could see the point everybody they were trying to make. But at the end of the day I’m like, I don’t think it matters what we call it. Right. We know what this pipeline does. Maybe the term isn’t perfect, I don’t think. I mean with as much arguing as we did on that thread, I don’t think there isn’t one term that can rule them. Also, does it really matter? And I’m kind of busy, so I gave in. I’m torn because I get a turbot phrases are important. There’s power in words. But again, yeah, sometimes what’s important to the idea.

Mike Pfeiffer:
Yeah. Well I think what you shared there is a good thing for people to ponder because not everybody’s had the reps to go through that. Right. And so I think that your experience or a lot of value, yeah. The last thing I’d like to ask you is people listening to this show that don’t work in a team that actually developed software might be wondering, do I even need to care about DevOps at all? So I love this hear what you think from that perspective for a team that they’re just doing infrastructure. Do they need to start looking at this concept and thinking about this more or is it really just for people building apps and writing their own software?

Abel Wang:
I think it’s absolutely for infrastructure for people as well because ultimately if you look at an app, an app consists of two parts, right? There’s the code and there’s also the infrastructure that the code runs on. And this is fun for me to talk about because I am a pure software developer, right? I’ve been writing code for, I don’t know, a zillion years by now. It’s what I do. It’s what I love. I’m all about code and I couldn’t give a rat’s ass about hardware. For the longest time I was like, other people can figure that stuff out because I just don’t care.

But the more that I’ve worked in the industry, the more I realize that the infrastructure that your app runs on is every bit as important as the beautiful source code that I’m writing, right. In my mind, it’s gorgeous. Right?

So, and that was kind of interesting for my brain to reconcile to be like, yeah, the hardware or the infrastructure that runs on completely determines what happens with my code to the point where it really, really matters. Right? So if the ultimate goal is that we want to be able to deliver value to our end users continuously as quickly as possible, as much as we can, we have to pay attention to the infrastructure as well as the source code because the infrastructure is every bit as important. Why do I love infrastructure as code? And I can’t believe I’m using the term get ops and all that. And it’s because everything now is stored inside of source control, including your source code, your infrastructure, your database schemas, right? All that stuff can be stored inside of source code, your pipelines. That means everything gets versioned together.

Right? So I can go back to any point in time and be like, okay, at this point in time, here’s the source for my application. Here is the infrastructure for my application, here’s the database schema for my application and here’s the pipeline that will build and deploy it. Right? And you can just immediately do that for any point in time and everything gets versioned together. That makes me incredibly happy. Again, to wrap that up. Why is this so important for infrastructure people? Because the infrastructure is every bit as important as part of the code and if we want to continuously deliver value, we have to pay attention to both.

Mike Pfeiffer:
Totally agree. Abel Wang thank you so much for being on the show today. I appreciate you taking the time. I know you’re super busy. Thanks so much. 2020 can’t wait to see what you do out there this year.

Abel Wang:
Cool. Thanks for having me, man.

Mike Pfeiffer:
Thanks.

Abel Wang:
Yup.