About 3 years ago, I received an email from a network requesting a 30-minute Zoom meeting to discuss a project. For privacy purposes, I will call the person Steve. Our meeting extended to about 57 minutes. Steve had a good project idea and had already assembled a team of 5. He expressed the need for someone to lead the remote team and project.
I was curious and asked about the challenges they were facing. I discovered two significant issues:
1. Endless development cycle: The team was stuck in a continuous loop of development, acceptance, and changes as requirements became clearer.
2. Manual processes: All development and deployment processes were manual.
While they intended to use a non-waterfall, incremental approach, their practices resembled a disguised waterfall. This was evident in the numerous dependencies, lack of time-bound iterations, excessive meetings, fewer working features, and constant technical debt.
Although communication was valued, there were inconsistencies in timing and frequency, which hindered the project's progress.

The Project
The project involved creating and maintaining a mobile and web application solution. It was divided into two phases: development and maintenance. The same team was responsible for handling both phases.

The Team
The team was cross-functional, with each member specializing in different areas of the project. This, however, led to high dependencies among team members.
Despite the limited budget, Steve was determined to see this project through. I recognized the project's challenges but found it intriguing.
I believed that the primary issue wasn't the team's competence but rather the approach used for development and delivery. To address these challenges, I proposed a strategy combining Agile Scrum and DevOps.
Agile, more than just a methodology, is a mindset that embraces flexibility, adaptability, and continuous learning. DevOps, similarly, isn't just about technology but a culture that breaks down silos, eliminates waste, and streamlines the software delivery lifecycle.
Given the limited budget and resources, I felt that this combined approach could be a valuable solution to the project's challenges.

Before the Transformation
Development
The project initially followed a traditional, unstructured approach. Meetings were held as needed, led by Steve, who shared his vision with the development team.
Key Challenges:
Sequential Development: Documentation of the System’s requirements show all required features and interfaces was created.
A design blueprint of the entire system was designed by a frontend developer. The team operated in a sequential manner, with one phase (design) preceding the next (development).
Limited Collaboration and high dependencies: Sub-team A's work often stalled the progress of sub-team B. While all these activities were waged on a set of team members, the other team members were waiting for sub-team A to finish before a kickstart with sub-team B.
Lack of Agile Practices: The project lacked the iterative and incremental approach of Agile methodologies. In fact, the entire designed system was planned, designed, developed before testing took place.

Technical Debt: A significant amount of technical debt accumulated due to delayed testing and compromised quality.
Evolving Requirements: The project faced challenges adapting to changing requirements.

Specific Issues:
Unplanned Meetings: Meetings were held upon requests, without a structured approach (conveyed by Steve).
Manual Processes: The team likely relied heavily on manual processes, which can be time-consuming and error-prone.
Lack of Testing: The focus on development without adequate testing led to quality issues and technical debt.
This pushed the team back into fix bugs of the entire system and sub-system (too much waste). This cycle repeated severally even as requirements evolved.
Overall, the project faced challenges due to a lack of structure, collaboration, and quality assurance. These issues contributed to delays, increased costs, and a low-quality product.
After a serious struggle, the first version of the app was launched for a pilot test.

Deployment
The team deployed the application using a traditional approach, manually pushing and cloning Git repositories to a manually created and provisioned virtual machine. Whenever an update was made, a developer would manually log in to the system and perform a Git clone or git pull.

Agile Transformation
Scrum was introduced, establishing a strong framework for project management. The project was divided into smaller, manageable units, treated as individual projects, and subjected to time-bound iterations of various processes. This fostered continuous improvement, constant communication, and feedback. Items were prioritized based on their impact on the project outcome, and the team became self-managed with minimal coordination from a single individual. Project management software like OrangeScrum and Notion was used to maintain a centralized source of truth.
Developers took ownership of different features, reducing dependencies, waste, and improving work efficiency and focus. Daily stand-ups helped identify and resolve issues early, while feedback was received promptly. These factors contributed to a more organized and productive team.
Features that once took months to deliver were now completed as functional units within weeks, without compromising quality. Development seamlessly integrated with maintenance (operations) thanks to the introduction of DevOps.

DevOps Transformation
Manual tasks were automated by leveraging powerful DevOps technologies. As a dedicated advocate for DevOps practices, I personally implemented containerization, CI/CD pipelines, infrastructure automation, test automation, and GitOps within the project.
Our technology stack included GitHub Actions, Docker, AWS ECR, AWS Beanstalk, Docker Compose, and GitHub. This combination of tools streamlined the entire development and delivery process, making it more efficient and enjoyable.

DevOps is extremely powerful, it played a crucial role in our project's success. By automating tasks and leveraging powerful tools, we significantly improved efficiency and quality. Our technology stack at the beginning of implementation included GitHub Actions, Docker, AWS ECR, AWS Beanstalk, Docker Compose, and GitHub.
While we didn't fully utilize the full potential of DevOps, we achieved substantial benefits. We implemented containerization, CI/CD pipelines, infrastructure automation, test automation, and GitOps. This led to higher-quality software, faster time-to-market, and improved development efficiency.
A once-dispirited team became self-managed and motivated, boosting performance by over 50% (I am not kidding). This transformation was possible due to the effective implementation of Agile and DevOps practices.

Part Two: Building a Resilient, Secure, and Scalable Solution (Coming soon!)
In the second part, I will dig deep into the specific technologies and strategies we used to create a robust and reliable infrastructure. We leveraged AWS Elastic Container Service (ECS) with auto-scaling, load balancing, deployment automation, and rollback capabilities to ensure minimal downtime.
I will also discuss the security measures we implemented, such as configurations/restrictions, access controls, and vulnerability scanning. Additionally, we focused on cost optimization by utilizing cost-effective AWS services and implementing best practices.
By sharing these details, I aim to provide valuable insights into building a successful cloud-native application using DevOps principles.

Conclusion
In this article, I have shared my experience leading an Agile and DevOps transformation in a project with limited resources and a legacy approach.
By adopting Agile (Scrum) methodologies and DevOps practices and leveraging AWS, we were able to successfully transform the project, despite the initial challenges. This transformation demonstrated that even with limited resources, it's possible to achieve significant improvements in development and delivery pipeline.