helmfile: difference between sync and apply (Helm 3)

Helm and helmfile are great tools to automate kubernetes deployments. However, they have some subtleties that are sometimes hard to understand and may lead to catastrophic problems. One of them is the difference between helmfile sync and helmfile apply, a question raised many times, for example in StackOverflow.

Running helmfile -h, the explanation of those two commands is:

sync → sync all resources from state file (repos, releases and chart deps)
apply → apply all resources from state file only when there are changes

But what does it mean exactly ? What are the differences and pitfalls ? Let's dive in together, starting at the basics of Helm 3 up to helmfile.

Note: if you are familiar with how Helm 3 upgrades work, you can skip directly to the last section.

Helm states

The first thing to understand is how Helm stores states.

Helm generates the Kubernetes manifests to apply to a Kubernetes cluster by "compiling" a Chart's templates against some values, that can come from the chart's values.yaml or the values override (defined in helmfile, passed using the --set option of the cli, etc.). All those information together (chart, values, options) are what we will call a state.

Whenever you install a release, Helm stores this state in a secret called (the v1 suffix being the revision):

sh.helm.release.v1.{RELEASE_NAME}.v1

This secret is simply a compressed, base64-encoded JSON stored in a single key - release -, which contains everything needed to reconstruct exactly the helm chart, and to re-apply it with exactly the same values to reconstruct the release.

It can be inspected using the following command (see this gist):

kubectl get secret sh.helm.release.v1.<RELEASE_NAME>.v<REV> \
  -o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -d

Content of the helm-release secret

If you decode this secret, you'll see a JSON that contains:

name, namespace, and version of the release
list of all chart files (name + base64 content of all files, excluding templates/*, Chart.yaml and values.yaml)
metadata (content of Chart.yaml)
values (content of values.yaml)
templates files (name + base64 content)
config (value overrides via cmd or helmfile)
values schema (content of values.schema.json)
hooks
info → current state of the release in Kubernetes (e.g. "install complete"), dates of first/last deployment, etc.
actual Kubernetes manifest (output of all rendered templates, this time in plain text)

When you upgrade a release, the new state is stored in a new secret, with the version incremented to the new revision:

sh.helm.release.v1.{RELEASE_NAME}.v{REVISION}

How helm 3 upgrade/rollback works

Now, let's understand how Helm decides what to do during an upgrade.

⚠️ Every time you run helm upgrade or helm rollback, a new revision (and secret) is always created, whether or not there are changes.

Helm 2 and two-way merge

Back in Helm 2, the upgrade process was plain and simple: helm reconstructed the old state by decoding the helm-release secret of the current revision, and compared it with the desired state to create the different patches to apply. This is known as two-way merge:

old state → desired state

The desired state can be reconstructed from a helm-release secret (rollback), or from a new version of the chart + values (upgrade).

The important point is that Helm 2 didn't take the live state into account, that is, what is effectively present in the cluster. In other words, if you modified anything manually (add a value to a ConfigMap, or a sidecar container in a deployment), this change was not seen at all by Helm, and could either be left as-is, disappear, or be overwritten depending on Helm old/desired states.

Helm 3 and three-way strategic merge

Helm 3 introduced a brand new way of computing patches required for upgrades and rollbacks, known as three-way strategic merge.
The article Three-way merging: A look under the hood, gives a good explanation of what three-way merging means (heavily used in git), while Helm doc's section Improved Upgrade Strategy: 3-way Strategic Merge Patches focuses more on what it means in Helm.

But simply put, Helm 3 now takes the live state into account:

(old state → desired state) → (live state → desired state)

The rules are (fields = key+value):

+ (add) → new fields in the desired state not present in the old state are added (overwriting any live state)
⌫ (remove) → fields existing in the old state that are not present in the desired state are removed (even if their value changed in the live state)
± (overwrite) → fields in the live state that are also present in the desired state but have a different value are updated (whatever the old state)
∅ (ignore) → the rest is left unchanged (e.g. new fields in the live state stay)

Moreover, the patches do merge operations, meaning maps are deep-merged (vs completely replaced). This is a huge improvement from Helm 2. Among others, it means that in Helm 3:

it is possible to add a sidecar container to a deployment manually, or a new data entry in a ConfigMap. If they are not managed by Helm (no fields in the generated manifests about it), they will stay unchanged after upgrade/rollback;
if you modify fields managed by Helm manually, doing a rollback will effectively reset the fields to the Helm values.

Simple example

Let's say you install a deployment with Helm with the following labels (manifest stripped for readability):

apiVersion: apps/v1
kind: Deployment
# ...
spec:
  # ...
  template:
    # ...
    metadata:
      labels:
        label-1: install
        label-2: install
        label-3: install
    # ...

Now, you change the labels manually to:

labels:
  label-1: manual
  label-2: manual
  label-3: manual
  new-one: added   # also add one

And finally do a Helm upgrade, with the new values being:

labels:
  label-1: install  # no change
  label-2: upgrade  # change
                    # delete
  label-4: upgrade  # add

The result will be:

labels:
  label-1: install  # overwritten
  label-2: upgrade  # overwritten
                    # deleted
  label-4: upgrade  # added
  new-one: added    # ignored/kept

Helmfile: sync vs apply

Now that we understand how helm upgrades work, let's dive into helmfile sync vs apply.

The helmfile sync command will run helm upgrade on all releases. This means all releases will have their revision incremented by one. However, as Helm does three-way strategic merges, if there is no change between the live and desired state, no patch will actually be applied: there is just a new helm-release secret created.

The helmfile apply command will run helm upgrade only when there are changes. However, to detect changes, helmfile uses the helm-diff plugin, which only computes the difference between the old vs desired state; it doesn't look at the live state (similar to Helm 2). If something changed outside of Helm, helm-diff will return "no change", and the release won't be upgraded.

In other words, apply has the advantage of not creating useless new revisions, but doesn't guarantee the coherency of the live state, as manual changes may go undetected. sync is exactly the opposite: it always creates new revisions for all releases, but will detect and undo any manual change that happened on Helm-managed fields.

UPDATE (May 2022): helm-diff added support for three way merge diffs on v3.3.0 (January 10, 2022). As the helm-diff process inherits environment from the helmfile process, one can do:

# use three-way merge strategy for diffing
HELM_DIFF_THREE_WAY_MERGE=true helmfile apply

This way, `helmfile apply` can now detect and change manual changes as well.

sync or apply is thus down to a trade-off, which is alleviated if you decide to never change anything manually (or always use three-way merge diffs). If you stick with this best practice, apply is always the way to go.

Written with ❤ by derlin