Introduction

Hello, I want to discuss with you today a problem I recently encountered with the microservices architecture and the solution I found. I hope my experience with this matter will help someone in the future or save some time.

This article is relevant if you use a container orchestration tool like Kubernetes or Openshift and manage the releases using Helm. However, it can also be a starting point for an idea in multiple areas.

These concepts are familiar, and I have not invented them; I combined them into a solution that works for the current situation of the project I am working on.

With this in mind, let’s start!

The problem

Microservice architecture is great for scalability, development speed, and making your solution technology agnostic by allowing each team to develop their service in the preferred technology. It also leverages the service's ownership to the development team that created it.

Even if everything sounds great, a problem arises: engineers are managing the Helm Charts that are the bases for the releases that will be installed in the clusters.

Usually, the main focus of the engineers is to make the most efficient code possible, so the focus won’t be on having the best Helm templates or configurations.

It’s also usually impossible in a large ecosystem to have a DevOps engineer dedicated to each team managing a microservice that ensures the best possible configuration and best practices.

This combination of factors is generating a situation where even if 80% of all services are using the same templates (e.g. Deployments, Services, etc.), all of them are written differently, the best practices are not respected, and there might be code duplication or even unused code because of the copy-paste pattern that is happening when a new service is created (this can be avoided using a template repository). Also, if you need to change something on all microservices, you must work in multiple repositories.

As a DevOps, working in this setup is like speaking a new language with each microservice.

The Solution

A centralised helm chart with template definitions that can be added as a dependency in all microservice charts sounds like a great idea. This way, the engineers are left only to configure the templates with the necessary values based on the service specification and decide what Kubernetes resources they need.

Also, when there is a need to update the configuration for a specific resource, e.g. adding affinities, tolerations, etc., a DevOps engineer will need to do the change in a single place and using the versioning feature will be able to propagate the change to all Releases.

To integrate the central charts into the microservice one, we can use the dependency functionality:

dependencies:
- name: unified-templates
  version: 1.0.0
  repository: <repository_where_chart_is_stored>

Implementation

I created a new repository and initialised a Helm chart using the helm create command to implement this.

Regarding the way the templates are imported, I decided to create functions. It is easier to track what objects are in use on a microservice, and you can easily template the object with the local values scope.

## Defining a template
{{ define "deployment" }}
api: v1
kind: Deployment
metadata:
  name: {{ include "unified-helm-template.fullname" . }}
spec:
  {{- with .Values.deployment }}
  replicas: {{ .replicas }}
  {{- end }}
{{ end }}

## Importing a template
{{ include "deployment" . }}

## Configuring the chart
fullnameOverride: <service_release_name>

## Configuring a template
deployment:
 replicas: 2

For the helpers functions to work correctly, since the templating will be done in the service chart, we need to define the fullnameOverride variable to match the release name. Like this, we can use in the unified charts the predefined helpers, which will use the override value to generate proper naming and labels.

To configure the template in the child chart, an engineer needs to define an object matching the name of the functions under which it will configure the required values.

Overriding

Sometimes, you must implement custom functionality, add some logic, or extend the template's functionality quickly. For this situation, I don’t want to restrict the user with the initial implementation, but it would be cool to allow them to override it.

To achieve this, I added a change to how the template is defined. Now, you can merge a second function with the initial one, obtaining the override functionality.

## Template definition with override logic
{{- define "deployment" -}}
    {{- $override := include "deploymentOverride" . | fromYaml -}}
    {{- $base := include "deploymentInstance" . | fromYaml -}}
    {{- if $override -}}
        {{- $merged := mustMergeOverwrite (dict) $base $override -}}
        {{- toYaml $merged -}}
    {{- else -}}
        {{- toYaml $base -}}
    {{- end -}}
{{- end -}}

## Importing the template
{{ include "deployment" .}}

## Override for custom use case
{{ define "deploymentOverride" }}
spec:
 replicas: {{ ternary true false (eq .Values.customVar "true") }}
{{ end }}

Documentation

Good documentation for the templates' configurations is essential for centralising them. To manage this, I used a tool to generate markdown documentation from Helm value files.

# -- Object that configures Deployment instance
deployment:
  # -- Define the minimum number of seconds for which the pod should be running without crushing before being considered healthy
  minReadySeconds:
  # -- Decide if the pods should not be restarted if you have secrets used as env vars and they are updated
  skipSyncSecrets: true
  # -- Decide if the pods should be restarted if you have configs used as env vars and they are updated
  skipSyncConfigs: true
  # -- Update strategy
  strategy:
  # -- Number of replicas that will be created

To automate the process of creating and updating the documentation, I added the execution of this tool as a pre-commit hook so that every time someone contributes to the repo, the documentation will be automatically updated:

repos:
  - repo: <https://github.com/norwoodj/helm-docs>
    rev: v1.2.0
    hooks:
      - id: helm-docs
        args:
          # Make the tool search for charts only under the `unified-templates` directory
          - --chart-search-root=unified-templates

Unit Testing

I needed a way to test the templates I had written before exposing them to the engineers, so I used the helm-unittest plugin to write some unit tests. I know it’s not the perfect way to test functionality, but we are assured that the templating works as desired.

suite: Deployment tests
templates:
  - deployment/deployment.yaml
tests:
  - it: renders a valid deployment instance resource
    values:
    - ./values/deployment.yaml
    asserts:
    - isKind:
        of: Deployment
    - matchSnapshot: {}
  - it: configures the deployment name correctly
    values:
    - ./values/deployment.yaml
    asserts:
    - equal:
        path: metadata.name
        value: service-template

Conclusions

There is a fine line between a guideline and a rule, between adding some generic best practices and forcing someone to adapt to a particular style. A good engineer has to develop solutions that consider all of these things.

The solution described above combines all of them, providing a guideline and some strict rules but also allowing for adaptation when necessary.

What is good for me might be bad for you, and as a DevOps, you have to consider everyone’s well-being, from client to engineer. It’s hard at times, but there lies the beauty.

You can find a template of this solution here.