There are two main tools to help parameterise Kubernetes application deployments, Helm and Kustomize. While Helm focuses on allowing you to build or use others' applications based on parameterising manifests using templating, Kustomize avoids templating by allowing you to create overlays that can override a base configuration for specific deployments.

While the two can be used independently, they are more powerful when used together. The approach I use the most is to use Kustomize to declare your own custom app deployments and use Helm to deploy ready made deployments of 3rd party services.

However, I do run into a limitation of Kustomize frequently for larger, multi-app deployments, namely, repetition.

Frequently we have parameters that we would like to declare at a global level and inject them somehow into our various apps within our deployment. Think the hostname for a platform, which is shared by all of our app ingresses. If the hostname changes, we need to find and replace it in all of our ingresses. This can be accomplished with other Kustomize functionality (replacements), which will be illustrated further down, but as we will see, this can only take us so far.

Ultimately, the feature we are seeking is templating. Kustomize offers no templating option of its own, and the Kustomize maintainers have clearly stated that they do not plan to provide it. It is such a frequently requested feature that the Kustomize maintainers now require you to tick a box on feature requests stating that the feature you are requesting does not involve templating. This may be because Kustomize has declared itself as the "template free configuration tool" for Kubernetes, which is fine. But with so many requesting the feature it leaves us with the question, how can we solve this problem on our own?

I would like to share the approach we have taken, as I think it would be of use. Firstly though, lets recap the built-in features that Kustomize offers to address duplication amongst manfiests. It may well be that your use case is already supported by Kustomize.

The Example Repo

I have created a repository to illustrate the problem and make the solution clearer. You can find it at https://github.com/steg87/kustomize-templating.

It is a very simple example, with two apps serving different (but very similar) html files using nginx containers.

The different milestones in this blog post are tagged in git.

There is a README.md included in the root of the repo to help you get set up with a demonstration deployment if you want to play about with it. The README.md instructions are updated in line with the tagged versions, so revisit it when you move on. In particular, the commands to build and deploy the platform are given in the makefile.

The Problem

The example starts with v0.1.0. The issue with this version is that we have our platform host define in two different ingresses.

# app1/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app1
  annotations:
    # Remove the /app1 prefix from path when passing to upstream service
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx # Ingress class for default minikube ingress controller
  rules:
    - host: local.minikube.com
      http:
        paths:
          - path: /app1
            pathType: Prefix
            backend:
              service:
                name: app1
                port:
                  name: http

# app2/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app2
  annotations:
    # Remove the /app2 prefix from path when passing to upstream service
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx # Ingress class for default minikube ingress controller
  rules:
    - host: local.minikube.com
      http:
        paths:
          - path: /app2
            pathType: Prefix
            backend:
              service:
                name: app2
                port:
                  name: http

If we need to update platform host, then we need to do so in both of the ingresses. While not a big deal in this deployment, if we have tens of apps it can be frustrating.

In the first instance, we can resolve this with Kustomize Replacements.

Kustomize Replacements

To allow for global parameters, we can use Kustomize Replacements. Replacements allow us to replace a target manifest field with a source value. Source values can be taken from any manifest generated as part of the Kustomize built, but often a ConfigMap is the best choice. In this ConfigMap we can declare our global variables at the root level (or wherever you choose, the point is that they are not local to any individual app).

To follow along with the example repo, checkout v0.1.1.

# config/globals.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: globals
  annotations:
    # Tells Kustomize not to output this manifest as part of the build. It is
    # only used for local configuration.
    config.kubernetes.io/local-config: "true"
data:
  hostname: local.minikube.com

As we can see, we have added hostname: local.minikube.com as a global variable, which is now available for replacement into the apps.

Before we proceed, let's remove the hardcoded hostnames from our ingresses. I like to declare values that should be replaced with <<REPLACE>>. This can indicate to a developer that the value will be generated at build time. It's also handy to use a constant, known string to indicate replacements as Kustomize will not fail if you accidentally replace the wrong field. Instead it will overwrite that field and leave the indended one with its original value. I always test my builds locally by redirecting build output to a file (with make build > k.yaml) and search for <<REPLACE>> to check if any replacements have been misconfigured. Updating this value will also help convince you that replacements work, as there's not much point in replacing a field with the same value.

Next, we need to map this replacement parameter to fields in our apps. We do this from the Kustomization file of our app.

# app1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: app1

resources:
  - ../config # We need to include the config map from the config directory
  - namespace.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml

# This generator just generates a config map from the local html file so we can
# mount it into the deployment to be served by nginx
configMapGenerator:
  - name: app1-index
    files:
      - index.html=files/index.html
    options:
      disableNameSuffixHash: true

# Configure the replacements for the deployment
replacements:
  - source:
      kind: ConfigMap
      name: globals
      fieldPath: data.hostname
    targets:
      - select:
          kind: Ingress
          name: app1
        fieldPaths:
          - spec.rules.0.host

Firstly, note that we have to include the ./config directory as part of the app Kustomization resources. This gives Kustomize access to the global parameters.

Next, we add a replacements section, mapping the source parameter (or globals configmap data field) to a target field (the ingress hostname).

Note how paths are defined:

• Subfields are delimited by '.'
• Indices are input as subfields, zero indexed as you would expect

From make build we can see the output is exactly the same as before.

You can also replace only part of a field value, if you can configure the delimiters to allow that. I won't go into detail on that, but you can read more in the replacement docs, see "Delimiter" and "Index" sections. But this is about as far as replacements can take us. We cannot replace anything that isn't a Kubernetes resource field. To see an illustration of this point, read on.

Templating

We will move on to implementing templating in the worked example shortly. It is as simplified as possible, but demonstrates the point. First, to give more context, I will discuss a real world example.

Real World Problem

The original reason I had to implement this was AWS ACK Controller Role resources. They require you to define the permission policies in string json form. Here is an AWS IAM role for Cert Manager.

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  name: CertManager-dev
  assumeRolePolicyDocument: >
    {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/ABCDEGHIJKLMNOPQSTUVWXYZ"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "oidc.eks.eu-west-2.amazonaws.com/id/ABCDEGHIJKLMNOPQSTUVWXYZ:sub": "system:serviceaccount:cert-manager:cert-manager"
                  }
              }
          }
      ]
    }
  inlinePolicies:
    CertManager: >
      {
        "Version": "2012-10-17",
        "Statement": [
            {
              "Effect": "Allow",
              "Action": "route53:GetChange",
              "Resource": "arn:aws:route53:::change/*"
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets"
              ],
              "Resource": [
                "arn:aws:route53:::hostedzone/Z0123456789ABCDEFG"
              ]
            }
        ]
      }

We needed to change the cluster OIDC provider in the trust policy between our deployments, because each cluster has its own. There is no way to do this with Kustomize alone.

Worked Example

Back to our example, which is tag v0.1.3 in the repo. Let's say we want to modify the header shown in each app based on the environment we deploy to. This not only demonstrates that we can modify arbitrary parameters in our deployments, but that it doesn't even have to be a Kubernetes resource.

We will need to modify some of the text in the files/index.html file in each app.

I was looking for a general Go templating style CLI that I could use as part of my build command. In (Gomplate)[https://docs.gomplate.ca/] I found just that. It is a very simple, yet very powerful tool giving us access to Go templating from the command line. It is very lightweight, has an (easy install)[https://docs.gomplate.ca/installing/] and minimal configuration required. That being said, there is some recommended configuration that I suggest, which I will cover.

You can see the syntax that Gomplate uses in their (docs)[https://docs.gomplate.ca/]. We're only going to use simple variable substitution but much more is possible. I would advise not to get too carried away though, part of the reason I favour Kustomize over Helm is that it is far more readable.

The default template syntax for variable substitution in Gomplate is {{.config.myVar}}. You pass the variable values in on the command line using gomplate -c config=config.yaml .... Note that -c config=config.yaml means your variables will be available under the {{.config.myVar}} path. For clarity, this would also work -c vars=config.yaml with {{.vars.myVar}}.

You can pass in templates (files containing template syntax) either by string (-i flag), by file (-f flag) or, as we will do, by stdin (kustomize build . | gomplate -c config=config.yaml). The makefile build command has been modified to reflect this. An additional command make template has also been added, which is handy to see un-rendered templates when you encounter a templating error with a line number reference.

Only one further configuration is required. The default delimiters of Gomplate templating are not great for our use case. If you try to assign a Kubernetes resource value to a template value, you will get an error.

# app1/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app1
  annotations:
    # Remove the /app1 prefix from path when passing to upstream service
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx # Ingress class for default minikube ingress controller
  rules:
    - host: {{.config.hostname}}  # this will error
      http:
        paths:
          - path: /app1
            pathType: Prefix
            backend:
              service:
                name: app1
                port:
                  name: http

This is because a Kubernetes resource value starting with '{' will not be interpreted as a string and will fail validation. For this reason, I modified the left delimiter of Gomplate to be '${{', borrowing from bash syntax and in the knowledge that values stating with '$' will be interpreted as strings.

One further modification I make, which is relevant when your Kustomize deployments use Helm charts as sources, is to move away from the '{{', '}}' delimiters, as Helm can erronously try to render these during Kustomize Helm chart inflation. Instead, I use '{[', ']}' (inner square brackets).

We can define these configurations changes in a .gomplate.yaml file in the root of the repo, which is where Gomplate will look for configuration by default.

# .gomplate.yaml
leftDelim: "${["
rightDelim: "]}"

Now, with the configuration and makefile commands updated, we are ready to modify our app files/index.html files. We also need to template out the host in the ingress files, and remove the references to replacements.

<!-- app1/files/index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>App 1</title>
  </head>
  <body>
    <h1>App 1 - ${[.config.env]}</h1>
  </body>
</html>

# app1/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app1
  annotations:
    # Remove the /app1 prefix from path when passing to upstream service
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx # Ingress class for default minikube ingress controller
  rules:
    - host: ${[.config.hostname]}
      http:
        paths:
          - path: /app1
            pathType: Prefix
            backend:
              service:
                name: app1
                port:
                  name: http

When we render the output with make build > k.yaml we can see that the files have been updated as we expect. We can deploy using make deploy and see in our dev environment that the header at http://local.minikube.com/app1 has been updated.

This has been a simple example, but hopefully you can see how we can extend this by using multiple config.yaml files for different deployments. When combined with the app specific configuration options that Kustomize provides, we have the ability to parameterise anything we want in our deployments.

Summary

To implemement deployment global parameters we can use Kustomize replacements to a point. If you require more complex templating features, you can implement Gomplate as part of your manifest build pipeline. The suggested implementation should not interfere with any existing Kustomize (or Helm, if you also use it) builds.