Originally published on Medium

Three years ago I joined the AT&T R&D center as a full-stack developer. I was welcomed directly into a brand new project, where I coincidentally witnessed a restructuring of the company's engineering practices. As I was getting my photo id and laptop, seating arrangements changed, the department rebranded, QAs were being trained as developers, and developers started assuming operational tasks.

During these three years we moved fast and morale was high, even though we had to assume new responsibilities and learn new skills, often within a short time period. We grew professionally while witnessing rewarding metrics from almost 18 million devices in production.

As the dust settled, I thought about the maxims and methodologies that allowed us to build a good, testable product, easy to refactor without breaking, and with a fast deployment to production. These are some of them.

Developers as Feature Owners

A developer is not just a stop in the moving assembly line of a product, but a feature artisan from conception to delivery. The developer should be given technological autonomy and a sense of ownership of the feature. This begins in developers themselves slicing a feature into tasks, and by contributing in all stages of the software lifecycle, including observation and maintenance. They know best how to tackle the feature's challenges and bring it to fruition.

Given autonomy we chose the latest technologies, versions, and conventions, which above all kept us engaged and motivated to continuously refactor.

Contribute in All Areas

After the developer built a feature, they are the most knowledgeable about it and can contribute in all its lifecycle across testing, deployment, and maintenance. The more they contribute, the more they'll understand the big picture and take ownership of features.

Beyond full-stack, a developer can contribute in operational tasks by managing cloud services, administering repositories, orchestrating clusters, writing scripts to automate changes, running deployments, writing libraries for CI tools, and creating metrics widgets and dashboards.

We might have a lot of learning resources available, but jumping onboard is what makes really us feel professional progress and accomplishment.

Infrastructure as Code

The way to give developers the power to contribute in all areas is by making everything code. Define all infrastructure as code, versioned and committed into the same repository as the application code, and give developers the credentials to work on it like all other code. This includes cloud resources, templates and configuration files for the Kubernetes cluster, the CI pipeline, build scripts.

We had a hard working DevOps team to support developers, but gradually we picked up many operational and infrastructure responsibilities.

Automate All Processes

Deployment from one environment to another should be automated, defined in code, never depending on someone's memory (and never before weekends). The same is true for updates and creation of services and databases, etc. This can be encouraged with multiple staging environments, where deployments from one to the other serve as practice and quality assurance.

Let the Pipeline and Application Code Co-Evolve

Development must begin with a good CI/CD pipeline, and developers should fix and improve it as their application's home. When developers work on the pipeline, they can find creative and elegant solutions to issues by tweaking the application, thus making it easier to build, test and deploy.

Our pipeline originated from a dedicated DevOps team which worked very closely with us, and committed its code into the application's repo. With time, our pipeline and application code co-evolved into ever greater robustness and synergy.

Pipelines are notoriously "antifragile," and keeping them stable can be frustrating. Fixing failures is part of the process. Developers learn through them to debug their apps and learn its quirks, avoiding similar issues in production.

Smash the Monolith

When the project was just a blueprint and a few PoCs, the architects defined an API for us. The project had many parties and teams, but we all talked a shared vocabulary. Each could test the API on their end, and decouple themselves by mocking it.

Splitting your app into components and your back-end into micro-services and serverless functions encourages decoupling, which reduces team dependencies. When in doubt, split. It's easier to merge components later than separate when conjoined. Think about creative ways to split your front end into separate deployments as well with micro-frontends.

Monolith No, Monorepo Yes

Ironically, although we decoupled many components, we still want to keep them together in source. We started with one repo for each microservice and for each client, but we found ourselves cloning them in one big folder and writing awkward scripts to link and build them together. As features spanned separate repositories, we had to open multiple pull requests which in turn triggered multiple builds.

This is what monorepo tools are designed to solve, we use Lerna for all our JavaScript repos, and we're never going back.

Bite-Sized Pull Requests, Short-Lived Branches

Some call it trunk-based development. It's easier and faster to merge changes when changes are small. Small Pull Requests are easier for the developer, the reviewers, and other developers pulling changes. Drastic code changes bode bugs, and often come with technical debt that no one wants to pay.

Knowing pull requests are a continuous work in progress helps in code reviews. Nevertheless all pull requests should include tests and feature toggles, to avoid technical debt.

In our case, we deleted the branch after merges. Before shipping, we would create a release branch with a version number and test it thoroughly. Nothing would get merged to it except bug fixes. After release we created a git tag for future reference.

Reduce Hoops, Shorten Feedback Loops

As the software development cycle advances, changes become harder and time estimations longer. What a developer can detect immediately in their local environment can take hours in the CI builds, or days once deployed, sometimes weeks.

Test as earlier and often as possible, use quality gates in the pipeline. The "merge" button was greyed out for us unless two reviewers approved it and the build returned green. This includes code analysis, lint, security scans, unit tests, end-to-end automation, mutation tests, etc. With coverage thresholds. All can also run locally, and most third-party code scanning services offer a CLI or IDE plugin.

Containerize

For micro-services, this is one of the best ways to shorten feedback loops. The runtime that works on the developer's machine is the one that gets tested by the CI/CD pipeline, and is eventually shipped to production in the form of a Kubernetes pods. If a developer says "it works in my machine" and they use a container, we can be confident that it'll behave the same way in production.

Backend developers should see the container as the application's deliverable unit, not the folders with compiled files within. While both backend and client developers can isolate their application's environment in a declarative way, and share that environment with the rest of the team. In one of our projects we used Nix for this purpose.

Gamify Team Excellence, We Like to Show Off

The gamification of the office TV dashboards was accidental. We displayed pipeline build results - which were either green or red - together with various quality metrics, organized by teams. It turned into a competition which also improved team cohesion. Teams also showcased their work at demos, which was an opportunity for all involved to take pride in their contributions and congratulate each other.

When we started working from home we lost the effect of shared office TV dashboard. Build results and metrics turned into individualized alerts and notifications, but the demos and the congratulations didn't change.

We're All Learners, We Make Rookie Mistakes

We all want to excel professionally, but this sometimes comes in the form of perfectionism. One of the subtle signs of perfectionism is the fear of under-performing in fields in which we're inexperienced. A seasoned Java developer may refrain from writing JavaScript, as they might not know its quirks and feel awkward. This harms your ability to contribute to all processes and see the big picture.

Forgive yourself and others and allow yourself to be a rookie again, we have your back.

Give Kudos, Show You Care

This can't be automated, but it can become second nature. It's a basic human need to be recognized and feel that one's work is meaningful. This feels true when busy managers join demos and give feedback. A short email with "the client loved this feature" goes a long way. Likewise, think about congratulating a team member or welcoming someone new.

All the automation scripting and pipeline fixing can be tedious and mechanical, but it's the people which make it humane, feeling recognized and meaningful at the end of the day.

Future

As a developer, I'd like to "commit" what works. But it's impossible to define "methodology as code" like infrastructure, it's a soft science involving people. Behavior cannot be programmed, but it can be measured, fostered and encouraged.

There are many ways to improve our current working practices. Some involving data-driven decision making, and some by adding even more automation, often in creative ways.

With data, we can research trends and predict targets to aid in time and effort estimations, identify bottlenecks, and find ways to further decouple dependencies, automate, and reduce feedback loops.

Recently, we improved pull request pickup time by automatically pinging the relevant reviewers. This reduced the nudging we had from each other to review our pull requests, while clicking on the message conveniently redirects us to review the code. The message even includes estimated review time based on the type of file changed and its changes.

This wasn't a script someone in our team wrote, but part of a dedicated project on its own, currently scaling up. The project is also working to combine all dashboard, metrics, scores, and quality scan results into one big-data framework, a project on which I'm currently participating. We aim to foster a common language for the organization, a language with actionable feedback for teams, hopefully resulting in a healthier software lifecycle, and developers motivated to give their best work.

Twitter: @ronenl