Designs like a Cloud Architect, Ships like a Developer, Operates like a Site Reliability Engineer My transformation in 2020, and what Agile/Embedded DevOps Engineer Model is all about

2020 is finally over and it has been a hell of ride for me and I guess for everybody around the globe. Although, it was not the best year in my life personally as I have suffered a great length of social isolation due to COVID and some unfortunate circumstances, I still manage to hold myself together, be productive and kept growing. I may not be at my happiest state, but I certainly didn't made this an excuse to stop myself from working extra hard to pursue my goals and dreams, while keeping up with my professional duties. I feel more resilient and confident as a result of all of it. Besides all the negative, career wise this year also felt like the most accomplished year for me.

I will try to keep this article focused on my transformation in 2020, and try to explain you what Agile/Embedded DevOps Engineer model/role is.

2020 Highlights

First, a short list of my major achievements:

• New role as an Agile/Embedded DevOps Engineer
• A great deal of improvement my technical skills in PowerShell, Azure Cloud, Azure DevOps, Docker, Infrastructure as Code, Automation, Monitoring, CI/CD and Automation.
• Opened my first personal blog/portfolio website.
• My first open source contribution to Azure quickstart repository.
• Published 6 posts and 6 DevOps projects in my website.
• My submission won the Azure Festive Tech Calendar Automation Deployment Hackathon

https://www.wesleyhaakman.org/festive-tech-hackathon/

• I had connected and engaged with a lot of Cloud/DevOps/SRE professionals and content creators of all kind around the world via social media, learnt from what they are sharing and try to share my knowledge wherever I can to pay forward what I have been receiving online.

So What is Agile/Embedded DevOps Engineer

First of all, the exact terminology that we actually use in my company is "Embedded DevOps Engineer", Agile is something I've come up with so it resonates with a larger audience. If you have never heard this concept, don't worry I had not too until December 2019.

It all happened during my initial meeting with my current company's Vice President of Operations. He asked me, what do you think your role will be here like? I told him that I would oversee all DevOps related aspects of the project and pretty much drive the project, take it to the finish line.

As an operations engineer this is usually what is expected of you, being the owner of a particular project or a solution and delivering it end to end. Having "Root” level access to all, being keeper of secrets and systems.

His answer was short, he simply denied this approach and started to explain to me they are going to be implementing a new model called "Embedded DevOps” where an "Embedded DevOps” engineer(s) is a directly assigned resource of a software delivery team. Moreover, the DevOps mentality will be expected and adopted by all members of the engineering teams. So it will be a shared duty amongst everybody to operate the services.

The embedded engineers will also be engaged in a central "DevOps Practice” team that they would allocate a certain amount of time per week to work on common goals and practices that should be promoted to the rest of the engineering teams.

Then he asked me, what do you think about it? I said great! Let’s do it. Fast forward 12 months and it was surely one of the best work experiences I had so far.

Making the best use of your DevOps Talent

Having a group of talented DevOps engineers in your company is great, but they are no good to anyone if they are isolated from the application development processes. If they are kept aside in yet another silo labeled as “DevOps”. Instead having them embedded in the software development teams, would make their knowledge, skills and their unique perspective available to the software project, from day one.

Increasing the engagement also allows for software and QA engineers to learn DevOps concepts. Likewise, as a DevOps engineer I feel like I have learnt a lot in terms of software development too.

This is possible because there is almost no resistance in terms of accepting change. This is very important, that is why as DevOps engineers we shouldn't have to drive the change and DevOps adoption. Instead it should be everyone’s desire inside an organization to implement and practice DevOps while developing, delivering and supporting their products. Hence, a DevOps engineer's duty is more on enabling and improving these practices. Providing solutions to problems, where engineering teams need assistance.

What about company wide DevOps Challenges

This is where DevOps practice comes into play. Having embedded DevOps engineers are great, but this fills their day with development and operations tasks for their assigned product(s).

All DevOps engineers embedded or within the central DevOps team, all of them join once a week for a catch-up type of meeting where the structure is fluid. It may include, planning, announcements, discussion points and so on. Then, DevOps engineers in groups of four or five, split to work on different larger scope projects.

These projects are problems that are concerning the whole company. These are problems where a larger collaboration is needed either for designing or implementing a solution to a particular problem.

Key Requirements of the Agile/Embedded DevOps Model

Below are my key highlights of the necessary practices that needs be in place for a DevOps practice that works in this fashion:

• DevOps from the beginning, designing 12 factor apps means our projects are suited for faster change delivery. We are able to deliver a change in a matter of hours if needed.

• Security starts from code; with strong collaboration between security, software and DevOps engineers, source code, infrastructure and our application architecture is secured and hardened according to industry best practice starting from the development phase.

• Proactive and pre-planned work; this model allows for better allocation of DevOps engineer’s time in terms of task planning. They are clear of their responsibilities and priorities. This allows for better planning for their managers and themselves. This is a much better allocation compared to reactive and random project assignments that leaves DevOps engineers overwhelmed and alienated.

• Collaboration and transparency is encouraged across the company. Inside the company, information is free to flow, so everyone can learn. Repositories, Wikis, document sites are there for us to work better and find solutions faster.

• Sharing meetings, demos and similar workshops are regularly held.

• On-demand training resources are easily available and bootcamp type more structured training is offered on a regular schedule.

• Using Infrastructure as Code and end to end Automated Continuous Delivery.

Design Like an Architect - Your Personal Cloud SME

One my of duties is, acting as an immediately reachable Cloud SME(Subject Matter Expert), to my team and managers. Being a cloud enthusiast, this is one of the areas I excel at, and I have great pleasure when I have to work with this nature of a task or a story.

This year, my role allowed me to curate and extend cloud infrastructure for both brownfield and greenfield services either we host as a part of our customer facing products, or a line of business service we are hosting internally, that we use for business intelligence or software development operations.

Taking part in these major decisions is very important as the decisions made at this stage can make or break your solution's:

• Usability
• Reliability
• Scalability
• Availability
• Testability
• Supportability
• Security
• Liability

Which are common features for almost all enterprise software we use day to day. No matter how good a software product is, it won't be successful without having the features listed above. If you have frequent outages, or you can't figure out your products scaling bottleneck which ends up in poor performance, it probably won't attract a lot of market value. These are now expected as a default baseline by software customers and DevOps really shines assisting in delivery of these features.

Ship Like a Developer - Scrum for Ops

One of the main differences in this working style is that, my work is also planned in 2 week scrum sprints. This forced me into breaking down my stories and tasks, have a plan and a list of deliverables, acceptance criteria in a well documented manner that works for us. Effort estimations were difficult at first but, the more you do it the easier it gets.

My work is reviewed and merged via Pull Requests and rest of the team is familiarized and kept up to date of the process during Scrum meetings.Working closely with the Application Developers, UI/UX and Product Management allowed us to have a much more efficient SDLC processes overall that boosted up everybody's effectiveness.

It's not my character to contain myself inside a box, so I usually overreached and gone beyond my duties in understanding business logic of the services I worked on, which in return presented itself as enhanced value to the team and the products we are shipping.

Another significant benefit of working inside a development team is, knowledge sharing. I have certainly improved my programming skills, Database/Query language skills, CI/CD skills for different technology and infrastructure stacks and overall my comfort level while bouncing ideas of other Developers. I'm also sure that on the other hand my teammates after working with me now have a better understanding in cloud infrastructure, networking and overall software operations.

Operate Like a SRE (Site Reliability Engineer) - Modern SysAdmin in NoOps Era

Coding and releasing infrastructure is great and fun, but the story doesn't usually ends there. A developer too can most likely put together some infrastructure as code or automation to bring up infrastructure for their projects, but performing as a SRE requires a unique set of skills and mindset which cannot be overlooked.

While developing an application, in conjunction we are also developing code/config that provides integration with monitoring and observability tooling for alerts and visibility, configuration management, secret management, release management and more.

Deployment patterns like blue-green deployments for stateless services, schema updates to your database layers these all require some unique expertise to be planned, automated and performed.

I use to be a system administrator that does the same for infrastructure of our own, but as more and more compute is being passed on the cloud service provider's "SysAdmins" we get to operate our "Services" instead of "Infrastructure".

So, we won't be concerning ourselves, for hosting a highly available web server or a database server, PaaS offerings already gives us much of these, but instead the overall harmony of our micro services, their integration, the SLA's and SKUs we require to pay in order to fulfil our solution's requirement. This does not mean we can't, if need rises we can but we simply choose not to whenever possible as it all adds up as toils.

We also don't concern ourselves by hosting our CI/CD tooling, source control or any other collaboration tool and opt-in for cloud native SaaS and PaaS offerings wherever possible. Anything that adds an admin effort is actually more costly to a business. Your customers don't care about how well you host your Jenkins clusters or Octopus Deploy server, all the value is in the delivered software so the focus should be on those tasks, not things where there is someone else doing it as a commercial product.

All of this is also coherent with the designing element of this role. You can think about the whole picture, including the operations duties while making the architecture choices for your application/infrastructure.

Knowing a service by it's nature is very important in it's uptime, health and reliability. For a messaging infrastructure like a Kafka server or a object storage service like Azure Storage Accounts, you may be only needing do a metrics based monitoring but what about an API that is sitting behind a WAF and requires OAuth authentication, you cannot simply ping your API, you need to know your service in order to monitor it's health state and performance. This is were SRE skills comes into play that incorporates all of the supporting services, side by side with the actual application/service.

Conclusion

Overall, I can say that this year was another year of great deal of learning, accomplishment, recognition and action taking. I'm very proud of myself to go through this transformation and came out fine on the other side. I may be wronged in the future but I believe, our limits are defined by the mental ceilings we put for ourselves. I always actively try to set a higher ceiling and standards for myself. The only way to reach those ceilings is believing in myself that I will get there and put in the effort necessary, no excuses and by all means. I admit that I'm my own biggest fan and I suggest you should be yours too. If you don't believe in yourself, how can you expect others to do so.