Written by Joern Barthel
Published on April 13th 2021

An anecdote about adapting lessons from traditional engineering to the software industry

The software industry includes a lot of metaphors that stem from traditional engineering fields. What software engineers seem to lack, however is an appreciation for "good" standards and the slow but steady work of measuring and improving them. This practice seems to be front and center of well-known production systems such as the one Toyota uses. The question is: can we adopt some of them?

The software industry often does not feel very industrial. We call ourselves engineers. We’re doing Kanban or Scrum planning sessions. We incorporate and document lessons learned and (if we’re really mature) do continuous improvement processes of some sort. But is there something such as a general process to get good results? To iterate towards good results? Or is the process in essence “if you want to get good results, do it with good people”, aka no process at all?

In order to get a more lateral angle on this questions, I participated in a series of Toyota Production System workshops for engineers interested in an on-hands experience of the production processes at Toyota and it’s first and second tier suppliers. Needless to say, I am not a subject matter expert and this obviously taints the potential lessons I can take away from this experience. I nevertheless found some aspects quite instructive, specifically where they align, at least anecdotally, to our experience as consultants.

The most relevant take away is perhaps the customer focus that should not surprise anyone familiar with the customer obsession of organizations, such as e.g. Amazon Web Services. For these TPS practitioners quality is front and center because that is what customers demand: “Work is not something that you think is good enough; it’s that your customer is satisfied.” In their world, testing is also something that is never done by dedicated departments (that are detached from the actual work), but by the engineers themselves. The QA department is just picking random samples and publishing statistics, and adjusted by changes in process (if something changes, checks are increased).

Since quality is the driving force, the followup questions become: how can high quality be guaranteed? The TPS answer is creating and improving standards. Results are compared against the standard and either results or the standard receives improvement. And while this process is mandated from the very top of every organization, it is owned on the shop floor. In many cases the top management participates in improvement meetings once a month and thus makes sure that the process is alive and kicking.

The team lead's job is peculiar in this setup: they are usually not necessarily the best engineers but the best motivators. Their job is the integration of standards and keeping everyone on sticking to (as well as appreciating) the process. Enabling and making the time for continuous improvements requires defenders, and this is primary the team lead's job. The importance of this position is significant: team leads typically receive a lot of training in order to be able to fulfill their responsibilities. Continuous improvement processes never work “as is”: they need the attention of the top management and must be kept alive by those who work with them.

Standards and their continuous improvement have other benefits apart from quality as well. Cost for example generally tends to go up - cleaning up the waste of the non-value adding parts of the process improves the margins necessary to keep the lights on. Another impact is the creation of a blameless organization: if everyone sticks to the process, the people are never at fault. It also enables junior engineers to quickly get on board and last year's hires to train this year's hires.

The most crucial aspect of the continuous improvement cycle is the monitoring of its improvements. In general two aspects are emphasized: visual management and the quick availability of escalation paths. Quality and customer feedback is always made very visible, directly on the shop floor and fast feedback cycles are front and centre of every process. No one is rushing, but nobody's idle either? This is a strong visual cue (or business KPI) that things are in order for a given process.

In summary, people and standards drive the investment priorities of TPS shops. Obviously the people part (at least salary wise) applies to the software industry as well. But training and standards? And if training, for what specific standards? Are we as an industry ready to admit that not every project is a work of art that cannot be constrained by standards?

We at kreuzwerker believe that the answer is: yes.

One approach that we often recommend because of it's scope, ambition and general applicability is the AWS Well-Architected Framework. Since 2018 this framework has been at the core of the AWS set of engineering standards. In the beginning it covered mostly technical aspects, but has introduced more and more process and organizational issues over the years. The key features of the TPS as discussed in this anecdotal blog post are all there: fast feedback, observability, documentation, continuous improvement including technical patterns to archive fault tolerance or more secure delivery architectures. Together with the Well-Architected Review Tool workload owners and their executives can leverage Well-Architected Reviews in an iteration fashion. This enables teams and executives to understand where they are on the road to better quality and eventually to better customer outcomes.

Are you interested in learning how to leverage Well-Architected Review and the Well-Architected Review tool in your organization? And how it can benefit your transformation to a more agile organization?

Reach out to aws-leads@kreuzwerker.de - we’re happy to help.

Cover by Isis França on Unsplash