Written by Manuel Vogel
Originally published on April 7th 2021

We explain what it means to contribute to open-source software under the hood and to actively maintain such a project. In our case, the terraform-provider-docker.

The why for us

We all use open source projects to develop, run and evolve the software that we use both internally and at the clients who pay us for our expertise and experience. We find bugs, see the need to extend the functionality or stumble over a piece of documentation that is not fully clear and understandable. We know talented folks took precious time to design, create, maintain and evolve such open-source projects. As we are working on those projects heavily, we carve out time to make a contribution by creating bug reports and providing fixes for them; and if we see the need for a new feature, try our best to provide it. In our case, we wanted to deploy docker services into a docker swarm via terraform. As those resources were not supported at that time in 2018, we filed a colossal pull request, which led us to important learnings. We will talk about them later in detail in this post. First, let's explain how it evolved that we started maintaining this terraform provider.

How it evolved

In 2017, after the provider-split in terraform 0.10, the docker provider was more or less abandoned and not actively maintained. We started contributing small features and bug fixes so we could use them in our deployment pipelines. We got in touch with several core contributors and began asking questions about the internals, especially about extending terraform, and how to write providers correctly. At that time, the documentation was still a work in progress, and for edge cases, there were no examples in the documentation yet. We started filing pull requests to update the documentation for this.

We continued providing the needed features for docker swarm and service resources, such as config and secret. After fruitful conversations and discussions with the Hashicorp provider team, we got to maintain the provider. It was the first opportunity to actively contribute in such a way to an open-source project that we also use internally and at our clients. As the next steps, HashiCorp released the terraform registry and distributed the community-maintained provider out of the HashiCorp GitHub organization. We took the provider over to the kreuzwerker GitHub organization in December 2020 to have the building and release process under our complete control. But what does it mean to maintain an open-source project?

What it means to maintain an open-source project

It requires a lot of coordination and fruitful discussions with engineers all over the world. It can become very time-consuming. For kreuzwerker, regular maintaining and providing feedback has become essential to keep such projects alive. It is often challenging to make extra time in addition to regular client work, to fix bugs and implement feature requests from people using our open-source software. But we accept the challenge and try to be as active as possible.

We think an active community is the most valuable asset for an open-source project. But like the code itself, it can also be one of its most significant liabilities. A community that multiplies can get out of hand, making it more difficult to stay organized and maintain. We also noticed that questions and feature requests began queueing up. We try to cope with the issues, share the workload on multiple shoulders and provide feedback on requests close to its creation.

Communication is crucial; that's why we want to keep it public, even if we receive private requests through other channels such as @gophers/terraform-provider-docker in Slack. We encourage engineers to open an issue or use the recently released discussions feature from GitHub. The code of conduct helped us set the etiquette guidelines, how we want to work together, and which tone and politeness we expect.

Of course, we encourage contributing and opening issues, leading us to the problem of handling and tackling them properly. We do not do fully automated housekeeping of issues and pull requests yet. It is planned. We have found templates for issues and pull requests beneficial, so a specific structure needs to be filled out, and the information for tackling a bug is already provided. For example, the version of terraform and the provider. These helpers significantly increase the likelihood in getting a bug fixed or a feature implemented.

Another essential aspect is git commits. We ensure conventional commits and encourage small and concise commits to make it easy to revert something. We are of the opinion as well that a good commit message answers three questions:

• Why is this change necessary?
• How does it address the issue?
• What effects does the patch have?

Once we have established automation and quality gates such as linting, testing, and document spelling, we also are interested in keeping it. In the long term, the benefits of those quality gates simplify the contribution and review process. For example, folks can automatically see if their changes break existing functionality or if the documentation is broken. The review process starts when all checks are green.

The final words of this block also go to a vital aspect, which is expectations. Maintaining an open-source project can provoke some negative feelings that have proven to be a weight on some maintainers' shoulders. This difficulty may lead to burnout, and ultimately, the abandonment of the project. It's ok to pause and take a break. We keep a steady pace, are always open for contributions, and give the contributors rights and power to collaborate.

The benefits

When we started maintaining, our knowledge of Golang was not very matured. We also started developing knowledge about terraform. But step-by-step, we gained trust and knowledge also with valuable feedback from other folks. We were constantly learning new tweaks, elegant solutions, and different approaches to solve problems. Furthermore, we dug deeper into the internals of terraform and got into direct contact with the terraform core leads, and met them in person at conferences such as HashiConf in Amsterdam in the past three years. We learned how important it is to keep documentation up-to-date when you work in a remote team, which internally had only a little experience; and in contrast to HashiCorp, which started as a remote-first company. These benefits lead us to the point where we will go with our open-source strategy and maintainership.

Where we will go

Currently, we are working on an internal refactoring to clean up code from the past years, even before we started the maintainership. Due to the upgrade to terraform-sdk-v2, we have new abilities for logging, debugging, and testing in isolation. Also, tools for the generation of documentation will simplify the process and keep it up-to-date. But first of all, we want to fix the reported bugs and clarify if they are still present after the update. We also plan to add support for running docker behind a jump host and provision docker containers. Our most aspiring goal is to come close to the docker CLI as possible. To achieve this, we plan to review which code we can reuse and integrate. For example, this has already been happened in the past to allow the converging possibility for docker services. We plan to add the generation of the changelog as we enforce already conventional commits from angular. After talking about the path towards milestone 3.0, we wrap it up with the conclusion.

Conclusion

In this post, we shared lessons, learnings, and skills we needed to grow and maintain an open-source project: how to encourage the collaboration styles we prefer, how to communicate effectively with the team and contributors, how to improve productivity leveraging Git and GitHub, how to consistently keep good quality, what version numbers mean, how to build and publish releases, how to write useful documentation, and how to prioritize all these tasks.

Additionally, in our open-source projects, we develop technical skills, share our experience and knowledge, learn and train to work in a distributed team with guardrails, clear instructions, and automation through bots.

Final words and kudos will go out to Shunsuke Suzuki who helped us move the development forward. And Anton Babenko for being a sparring partner for discussions on many terraform-related topics regarding the provider's evolution. Also take a look at his weekly dosis of terraform. If you're keen on contributing, feel free to reach out in the discussions or open an issue or a pull request. We're happy to onboard you!