The word "template" has 3 different meanings in Argo Workflows. Argo is much easier to understand once you know the differences between "template tags," "unit-of-work templates," and "(Cluster)WorkflowTemplates".

1. Template tags

In an Argo Workflow manifest (or in an Argo CronWorkflow, WorkflowTemplate, or ClusterWorkflowTemplate), some fields may contain "template tags". A "template tag" starts with {{ and ends with }}.

There are two types of template tags: variable-substitution template tags and expression template tags.

1.1 Variable-substitution template tags

Variable-substitution template tags look like {{some.variable.here}}. When a Workflow runs, Argo will replace the template tag with the value of the variable. Different variables are available in different parts of a Workflow manifest.

Example:

image: "ubuntu:{{inputs.parameters.version}}"

1.2 Expression template tags

Expression template tags look like {{=some expression here}}. Expressions may also use variables. But instead of simply inserting the variable's value, Argo executes some expression here as expr code. Then the expression template tag gets replaces with the result of that execution.

Example:

count: "{{=inputs.parameters['some-number'] * 3}}"

(Notice that there's a slightly different way to access parameters by name if the parameter name contains a hyphen. expr would have treated the hyphen as a subtraction operator.)

2. Unit-of-work templates

The second use of the word "template" in Argo Workflows is to refer to a unit of work. These templates are located in the Workflow manifest (or CronWorkflow, WorkflowTemplate, or ClusterWorkflowTemplate manifest) under spec.templates.

Example (from coinflip example:

spec:
  templates:
  - name: coinflip
    steps:
    - - name: flip-coin
        template: flip-coin
    - - name: heads
        template: heads
        when: "{{steps.flip-coin.outputs.result}} == heads"
      - name: tails
        template: tails
        when: "{{steps.flip-coin.outputs.result}} == tails"
  - name: flip-coin
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import random
        result = "heads" if random.randint(0,1) == 0 else "tails"
        print(result)
  - name: heads
    container:
      image: alpine:3.6
      command: [sh, -c]
      args: ["echo \"it was heads\""]
  - name: tails
    container:
      image: alpine:3.6
      command: [sh, -c]
      args: ["echo \"it was tails\""]

Each item under spec.templates is a unit-of-work template (referred to in documentation simply as a "template".)

There are several template types. The example above contains two container templates, one script template, and a steps template (a type of composite template to execute the other templates in a certain order).

3. (Cluster)WorkflowTemplates

The third and final use of the word "template" in Argo Workflows is to refer to the WorkflowTemplate and ClusterWorkflowTemplate resources. A ClusterWorkflowTemplate is just a WorkflowTemplate that lives at the cluster level instead of a namespace. For the purposes of this description, they're effectively the same.

A WorkflowTemplate has two potential uses.

3.1 WorkflowTemplate as an invokable workflow

When a Workflow is installed on a cluster, the Argo Workflows controller picks it up and immediately starts running it. But sometimes you want a workflow available to be run without actually running it immediately.

You could store the Workflow in git. But then if you have multiple users, you have to rely on them to authenticate with git and pull the latest source whenever they need to run a workflow.

Instead, you could define a whole Workflow and change the kind field in the manifest to WorkflowTemplate. After it's applied to the cluster, anyone with cluster access can run kubectl submit --from workflowtemplate/your-workflow-template. There's no need for git access, and they'll always be running the latest version.

3.2 WorkflowTemplate as a template library

In this case, "template" means the unit-of-work template.

Suppose you have a script-type template that sends a Slack notification. You might want to use this in a lot of different Workflows. Rather than copying and pasting the same template YAML into every Workflow, you could reference a template which is defined in a WorkflowTemplate.

Example WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: slack-workflow-template
spec:
  templates:
  - name: slack-template
    inputs:
      parameters:
      - name: message
    container:
      image: my-slack-notification-image
      command: [send-to-slack]
      args: ["{{inputs.parameters.message}}"]

Example Workflow that uses a template from a WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: task-with-slack-notification-
spec:
  entrypoint: task
  templates:
  - name: task
    steps:
    # other steps here
    - - name: slack
        templateRef:
          name: slack-workflow-template
          template: slack-template
        arguments:
          parameters:
          - name: message
            value: "task is done"

Conclusion

The three uses of the word "template" in Argo Workflows can cause confusion. I recommend using the terms "tag template," "unit-of-work template," and "(Cluster)WorkflowTemplate" to help differentiate.