You don't have to do Agile to be agile

Introduction

The rise of Agile brought with it a huge business anchored in helping organisations become Agile. From certifications, to coaches, to books, and millions of articles preaching on how teams will perform 200 times better.
The idea of being able to just implement a given methodology by the book and get that kind of performance improvement is hard to resist. However, these approaches almost never yield the promised outcomes, because they neglect to account for the multiple factors that affect a particular team's performance[1]. The outcome of this broken promise has left us in a precarious situation, where management expects to wave the wand of Agile and get results, but is not prepared to invest in the changes required to achieve those goals.
The common scenario then is having a development team that performs some sort of Agile methodology, but the whole organisation around it does not. We expect that we will be able to deliver on time, on budget, on spec, and even be able to change our minds as to what that spec is and still get it. That is, a rigid organisation with a small Agile cog in the midst of it. A recipe for disillusion.

Current approaches

Several Agile methodologies have been developed and are being used by teams around the world. Perhaps the most popular of all is Scrum. So let's look into it, but most importantly, into how it's normally applied, and why it usually fails to deliver the desired outcome. The purpose of this article is not too deeply explain the Scrum methodology, but to reflect on the aspects that often fall short when implemented.
The main premise is to iterate on a regular cadence called Sprint. Sprints can be of any arbitrary length, though usually they are 2 to 4 weeks long. During the Sprint, different Agile Ceremonies are performed to achieve the Team's goals for it.
We start with a Sprint Planning, where we define the work that we are looking to do during the Sprint. We use the concept of Story Points and Velocity to Forecast how much work we might be able to achieve during the Sprint. And with that we set to work.
During the duration of the Sprint we keep track of our progress using the Stand Up ceremony, where we attempt to correct course and attend to unexpected situations as they arise. We also perform tasks such as Backlog Grooming, where we aim to gather detailed requirements and estimate them so we can include in future Sprint Planning sessions.
At the end of the Sprint we gather together to perform a Sprint Retrospective. Here we want to work on how we work. Expose those things that worked well for the team, as well as those that didn't, with the idea of creating a Continuous Improvement culture. The outcome of these Retrospectives are actions that we take to make the way we work better and so improve our performance as a Team.
This is what we expect to implement, and in return get those amazing promised performance gains. However, when we try to apply these principles in our particular situation, we suffer from pitfalls that hinder our performance. Let's look into them.
Overseeing this process we are supposed to have a Scrum Master, whose responsibility is to help and coach the Team to achieve its potential. However, this role is rarely performed by a person with the right skill set, and often given to a Project Manager to whom we ask to deliver projects on time and on budget. On occasions we have these Project Managers take on certifications to be trained as Scrum Masters, however the pressure of delivering what the higher manager requires often overcomes the needs of the Team.
The Team members themselves naturally gravitate to silos of expertise, resulting in a factory line kind of process, where handing over artefacts back and forth is the norm instead of the Team working together to achieve a common goal. It then becomes a struggle to pull together in the same direction, when everybody is more concerned in satisfying the bureaucracy of tools like Definition of Ready, Definition of Done, and the like.
Another major issue I have come across while working in Scrum Teams is the dichotomy of Commitment and Forecast. When the Team plans the Sprint they Forecast how many Story Points they are likely to be able to achieve, based on previous iterations. However, this is often taken as a Commitment of how much work they will in fact complete. Leaving the Team divided between those pushing for a Commitment of what needs to be delivered, usually the Project Manager and the Product Owner, and the rest of the Team that will then push back on the expected Commitment with concerns around topics like Technical Debt and Requirements not being Ready to be worked on.
As a result of these and other shortcomings while implementing these kinds of Agile methodologies, we end up with a broken promise of performance, and an unhappy and burned out Team.

Factors that affect process

These problems are not necessarily a shortcoming of Scrum, or Agile industry itself, but a failure to analyse and understand the different factors that affect the efficiency of processes used to develop software. The wish to be able to simply apply a methodology and reap the benefits, while not taking into account your individual circumstances, is what leads to these shortcomings.
In my experience I've found three main factors that should be taken into account when shaping a process. Those are: the nature of the business, the product or project, and the people involved in them. It is important to understand that these factors are not stationary, and though they have their own pace, they all change and evolve over time.
Understanding the nature of the business and its market is fundamental to guide the kind of process we need to establish. This applies to the stage in the life of the business itself, are you an early stage start-up or a 20 years old established business, as well as the nature of the market and customers you serve, individuals or organisations. It is important to strike the right balance of nimbleness and robustness so that you can help your business accomplish its goals. This factor is likely to affect the level of rigour in our quality assurance and amount of experimentation we can afford to undertake. Another important aspect in business is the commercial model and commitments made to clients, deadlines and agreements that must be met, as well as financial goals that impact the work of the product team.
Similarly, the stage of a product is also important. Is this a brand new experimental product, an established product that we are maintaining, or is this a project to drive innovation in an existing legacy product? These questions will have an important impact on the kind of process we require to succeed. The most likely impact of this factor in the process will be the cadence in which we deliver work, the horizon of planning, and the weight of accurate estimation of effort.
Finally, is the people in the team that will have the most impact in the shape of your process. Obvious aspects to consider are the size of your team, the mix of skills and experience levels, as well as the tasks you expect each individual to perform. But just as important are the personality, working style, and learning approaches. Cultivating a diverse and inclusive team is not an easy but rewarding task that will bring enormous benefits to your business.

Conclusion

It's not easy to implement a perfect, bulletproof process, but to have a chance at achieving success we need to start by acknowledging our particular situation and acting accordingly. Silver bullets do not exist, and can't be bought. While engaging in training and coaching from Agile practitioners can help, it will not yield the expected results if we don't make the effort of applying it to our unique circumstances.
The most important thing to understand is that this is a moving target. Companies, products, and teams change and evolve over time. What worked yesterday might not work today. So your process must also change and adapt to your new situation. And for that to be possible you must remain agile.

References

[1] Krutchen, P.K.: Contextualizing Agile Software Development, EuroSPI Conference, Grenoble, France, September, 2010