Yikes. Unfortunately it’s true. I broke my ankle doing something I took for granted; walking. As I sit here and write this I am still in recovery, but it’s given me a few interesting perspectives on DevOps. PSA - please do not go breaking your ankles, or anyone elses. I am not advocating for this. Please.

A broken process needs time to heal

When you get extremely well-practiced at a task, things become as routine as walking down the hall, until you trip holding a ping pong paddle while batting at stressballs. No? Just me?... Let’s talk about CI/CD as an example of this. Last year the entire routine for the developers and DevOps got flipped on its head. Devs went from pressing all the buttons to none of the buttons. What used to be routine suddenly became verboten as I and the rest of DevOps took on roles and tried to implement procedures with Dev management to make up for the void that had been created by the walls that were constructed to delineate the distinction of each team’s priorities.

In our case we knew what exactly we had broken: the process. This was very straight-forward to us, from coding, to QA, to Releases, everything was broken. We limped along like Batman post-Bane because of our indomitable wills. The builds moved in mysterious ways, and thus releases were fragmented. You couldn’t easily know what was in production without having to login to BitBucket and review the commit SHA and then review the commits (a major undertaking for any dev who was trying to release anything). Tickets were QA’d individually so it wasn’t always clear where or how they interacted with each other until later, introducing bugs and frequently resulting in hot deploys. It had been like this since I joined the company two and a half years ago, and we hadn't had a reason to change. Why would we? Everything worked, didn't it?

For the sake of brevity I won’t go further, but you get the idea. Like me, this whole CI/CD pipeline needed surgery. Although we have surgically performed the painful part and that is behind us, recovery will take weeks. Sometimes though, it takes longer. Just like with an injury, if you try to do too much at once with it before it’s completely healed you injure yourself again and hurt yourself worse, thus you can drag a 3 week recovery into 2 months, or in our case a 1 business quarter task into 2.

Of course, we had had a DevOps department for a couple of years but in my opinion we never really did DevOps as a company until recently. Since the foundation of the department in 2017, we had been in a constant state of firefighting mode that made it impossible to get ahead of the curve. It took 3.5 months to straighten out the CI/CD pipeline to a point that you could trust the setup across the board. I remember the times where I'd have to rebuild a single branch 3 times to get it between QA, UAT(formerly Staging), and finally Production. Worse yet, the environment and configuration settings weren't identical so every environment was setup differently. It took another 6 months to migrate all of the critical systems' deploy pipelines into Octopus Deploy so that the overhead of managing it would be lessened for all parties involved. We did all of this while accepting infrastructure requests and performing migrations to newer hardware in an effort to keep up with the evolving technologies at DealerOn. As I look back I realize that if we had attempted this DevOps transformation back then, that we would never have completed as much of the work that we have in the timeframes that I listed above. We simply were not ready then, but that is not true today.

The moral of my story is that DevOps isn't magic. Although there are times that I feel like an arch-wizard with a mastery of the arcane, don't be fooled. DevOps won't instantly transform your business, and it certainly isn't a switch where it is suddenly ON and everything works better. DevOps is a mindset and a set of practices, and they take time to implement and reiterate as you come up with cleaner and better ways to do the tasks. Anyone who tells you otherwise is selling something. Though I am fairly new to the DevOps scene, I had never truly considered the scale of how a DevOps transformation would affect the department and company. After all, it was not until we had broken our metaphorical ankles that we had really started to heal and improve...

At the time of publishing this article, I'm pleased to announce that my leg is almost completely healed, and I'll be walking normally again very soon.