This was originally posted here.

Why do I need a Software Engineering Process

I remember when I started as a developer, I used to get so bugged writing unit tests and filling documents. I was one of the developers in a team of hundreds of developers, building a mammoth software which needed a monster server to run. For even a few line of changes, I had to write lengthy discourse about what was the change, the thinking behind the change and how I believed this will not impact the performance of the application. Then I had to add unit tests, followed by a series of regression tests. And before even sending the change for review, had to run regression tests for my part of the application and smoke tests for the whole application. What an overkill !!! At least that's what I used to think. But later I realised how all those steps were a necessary evil, and understood the importance of the process when I had my first production bug 🤦

Building software is hard, if anyone think differently I would like you to reconsider. There are many good tools out there to make our life easy as developers, but sadly we are left on our own when it comes to using the tools. There are guidelines and documentation on how to use the tools, but there are very few guidelines on how to build software the right way. Most of us learn by trial and error, learning from our mistakes. But I want you to learn from my mistakes and maybe have less heartburn next time you build software.

What is Engineering Process

Building software is an engineering process. Though we don’t relate to it, but building software is equally complex and challenging as manufacturing a car or building a skyscraper.

Any engineering process needs a set of guidelines that needs to be followed while we engineer a solution. A solution engineered without sound principles is inconsistent and tends to be brittle over time.

Engineering Principles are ideas, rules, and concepts which guides our engineering process. Though they may seem like an overkill, but it pays in the long run. Think of Engineering Process as taking your car out of parking, you will make sure the path is clear, check your blind spots and move on to the road.

Below is a sample Engineering Process which works for me. Tweak it as per your needs.

Engineering Process that works for me

Step 1: I will Design my solution before Implementation

Design before you build. This is a known concept in every engineering trade but somehow this eludes us when it comes to Software Engineering.

While building a house, we will never allow our contractor to start building unless he gets the designs approved. A car company will never start a production line unless every stakeholder agrees on the features and approves the design. Then why should it be any different when it comes to Software Engineering.

Prototyping a concept or writing quick and dirty code to learn a new language / technology is fine as either one can be discarded without any cost. But when it comes to production code, we should exactly know what we are doing. You will never drill a hole in your living room till you are 100% sure about what you are going to do with that hole.

Another problem that arises when we jump from requirement to code is, when the code goes for review. A typical story takes 1 - 2 working days of an engineer’s time. When the code goes for peer review, the reviewer is expected to provide feedback in few hours time. So there is added onus on the reviewer to understand the requirement, understand the engineer’s solution / thought process, apply the same to code and finally give a verdict all in few minutes of time, as engineers are measured on the amount of code they write and not the code they review. If we spend time collectively coming to a design solution, during review the reviewer only has to verify the solution against the approved design. This will reduce the review time as well as the chances of missed out logic or corner cases in the review.

Every Feature / Bug fix needs to designed, and the designs must be approved by your peers.

Step 2: I will Document my solution before Development

As a product grows, the code base grows and becomes more complex. Growing teams add engineers and engineers add more and more code. After sometime looking at code written a year back looks like mangled mess 🍝

Another challenge with ever evolving products is “Tribal Knowledge”. Before information could be passed down in written format, all useful information will be converted to poems and designated people in the tribe will spend their life memorising them and the saga continues for generations. Even with the advancement of technology, evolving product still are crippled by this problem. Every product has a go to guy / gal, who has all the poems memorised.

Only way to break this process is to document stuff. Not everyone is well versed in writing down their thoughts in a concise manner. Documentation doesn’t need to be written in fancy poetic English. Write documentation in the simplest consumable format. If words are too difficult to express the solution, use your drawing skills.

Every change to product must reflect in the living HLD (High Level Document).

Step 3: I will write code and explain

At the time of writing code, it makes perfect sense to us. The problem starts when revisit the same thing after sometime. We try looking at the code from all directions ⬆ ⬅ ↕ 🙃 😒 and start thinking why on earth did I write this and then we create a tech debt to clean up.

Continuous Improvement is good, but not every old code needs to be re-written. Knowing why you did something or why something needs to be changed will save you lot of heartburn and frustration 🤯

Get into the habit of writing Readme for every feature added or shortcuts taken which needs to be revisited. This will become the LLD (Low Level Document) of your code base which will benefit you or anyone who looks at it later.

Every change to code should reflect in the ReadMe of the project (Low Level Documentation).

Step 4: I will test my code

Whenever we buy any electronic good, most of them will have a small sticker which says “QC passed“.

All code is only as good as the tests which verifies it.

We all trust our code, as we have written it ourselves. But how does it hold good against the requirement? A function to add 2 numbers may compile and run fine in production, but it is of no value if the original requirement was to multiply 2 numbers 😊

Only unit testing your code shows your logic is solid and it works as expected. But code is part of an engineering solution which is expected to do a specific job. You can build the world’s best exhaust system but if it doesn’t fit in your final car’s model it is of no use. Hence it is important to test your code against the requirements for which it was written.

Every change must be Unit tested for logic, and Integration Tested for business requirements.

Step 5: I will track my changes

All effort put in engineering a solution is for a specific reason. We develop feature to be working in an expected manner. But how do we confirm if it is working as expected?

Product metrics are for measuring business success and cannot be a proxy for engineering success. A poorly engineered product can achieve business targets.

Every change we make is for a reason and the same reasoning must be reflected in tracking. If the change was engineered to improve the page load performance, we should be able to have enough tracking to prove we have improved the page load performance. If the feature release was done for user interaction, then we should have enough tracking to prove that the feature is working as expected.

Tracking should also be proactive in detecting system anomalies. It should inform the engineer when unexpected errors are occurring and collect relevant data to reproduce the error. It should also warn the engineer about possible system degradation (when user is having visible errors but performance is taking a hit). In a nutshell, every change should have enough metrics to give a detailed picture of the overall product.

Every change must define its tracking metrics and have enough trace to find anomalies.

Engineering Process is NOT a silver bullet which will solve all your problems. Having an Engineering Process will not make your bugs disappear or suddenly make your application run faster.

Engineering Process will bring sanity in your chaos filled process. It will help you build software products which can be reasoned, tracked and measured.