Kubernetes with Kubeadm: Fully Automated Installation with Terraform

Kubeadm is a Kubernetes distribution that provides all customization options that you can think of: container runtime, container network interface, cluster storage and ingress. You can configure all these aspects of your cluster but have to understand the individual options and their setup as well. For a complete overview about this remarkable distribution, see my previous article.

This article is a tutorial about creating a 3 node Kubernetes cluster. One node will be the control plane node, and 2 others will be the worker nodes. The particular Kubernetes components are etcd for the cluster state, container-d as the CRI, and calico as the CNI. Let’s get started.

The complete project source code is also available at my Github repository kubernetes-kubeadm-terraform.

This article originally appeared at my blog admantium.com.

Prerequisites

To follow along, you will need a Hetzner Cloud account. Then, get an access token following the official documentation.

Also, choose appropriate server types for your setup. For this demonstration, I will use the following:

1x CX11 node (1 Intel CPU, 2GB RAM, 40GB SSD)
2x CX21 nodes (2 Intel CPU, 4GB RAM, 40GB SSD)

You also need to install Terraform, head over to terraform.io and grab the binary suitable for your computer.

Project Structure

With the following command, you will create a file structure that conforms with Terraform best practices.

TF_PROJECT_NAME=kops_github

mkdir $TF_PROJECT_NAME

tffiles=('main' 'variables' 'providers' 'versions' 'outputs'); for file in "${tffiles[@]}" ; do touch "$TF_PROJECT_NAME/$file".tf; done

Provisioner Configuration

We start with configuring the used provides inside versions.tf. Enter the following content with the most up-to-date version that’s available:

//versions.tf
terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
            version = ">=1.35.2"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">=4.0.3"
    }
  }
}

The provider configuration is slim: We just need to define a variable that will keep the Hetzner cloud access token...

//providers.tf
provider "hcloud" {
  token = var.hcloud_token
  sensitive = true
}

... and the variable itself is defined in the same-named file.

//variables.tf
variable "hcloud_token" {
  sensitive = true
}

Secrets handling in Terraform means to not store them as concrete text because it will end up as plaintext in the state file. Instead, you pass them as environment variables to the terraform commands.

Resources

We will create two different kinds of resources: An SSH key for accessing the servers, and servers with the role of either controller or worker.

SSH Key

To access the machines, we will use a dynamically created SSH key. The resource configuration is this:

//resources.tf
resource "tls_private_key" "generic-ssh-key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

To use this key for connection to the server via SSH, we will use a local_provisioner, a Terraform abstraction that manifests resource information in files. We will grab the private and public key attributes from the resource and store them in files.

//resources.tf
resource "tls_private_key" "generic-ssh-key" {
  //...
  provisioner "local-exec" {
      command = <<EOF
        cat <<< "${tls_private_key.generic-ssh-key.private_key_openssh}" > .ssh/id_rsa.key
        cat <<<  "${tls_private_key.generic-ssh-key.public_key_openssh}" > .ssh/id_rsa.key
        chmod 400 .ssh/id_rsa.key
        chmod 400 .ssh/id_rsa.key
      EOF
  }
}

Server Variables

In Terraform, you can write resource definitions in plain text, or invoke functions that iterate through a list of items and use their values. My project uses the latter to be more flexible and handling any number of controller or worker nodes.

We need variables that define the list of controller and workers, and we need an object that holds the server’s configuration like server size and OS image.

// variables.tf
variable "server_config" {
  default = ({
    controller_server_type = "cx21"
    worker_server_type = "cpx21"
    image = "debian-11"
    k8s_controller_instances = ["controller"]
    k8s_worker_instances = ["worker1", "worker2"]
  })
}

Server Resources

I like to think of the following resource definitions as generators: They consume variables iteratively.

The controller is this:

resource "hcloud_server" "controller" {
  for_each    = toset(var.server_config.k8s_controller_instances)
  name        = each.key
  server_type = var.cloud_server_meta_config.server_type.controller
  image       = var.cloud_server_meta_config.image
  location    = "nbg1"
  ssh_keys    = [hcloud_ssh_key.primary-ssh-key.name]
}

And for the workers:

resource "hcloud_server" "worker" {
  for_each    = toset(var.server_config.worker_instances)
  name        = each.key
  server_type = var.server_config.worker_server_type
  image       = var.server_config.image
  location    = "fsn1"
  ssh_keys    = [hcloud_ssh_key.primary-ssh-key.name]
}

Kubeadm Automation: Controller

Up to this point, the above declarations will merely provision the infrastructure. In order to install kubeadm on them, we need to run shell scripts.

In Terraform, there is the concept of a provisioner, an additional logic that is involved during the creation of resources. We will use a remote_provisoner to copy installation scripts to the servers. This provisioner will use the SSH key that we created earlier.

The configuration inside the controller is this:

resource "hcloud_server" "controller" {
  //...

  connection {
    type        = "ssh"
    user        = "root"
    private_key = tls_private_key.generic-ssh-key.private_key_openssh
    host        = self.ipv4_address
  }

  provisioner "remote-exec" {
    scripts = [
      "./bin/01_install.sh",
      "./bin/02_kubeadm_init.sh"
    ]
  }
}

The two scripts will install all necessary packages and then start the kubeadm cluster initialization. The content of these scripts is derived from the manual installation tutorial, which will be published later.

When the controller is provisioned, we need to get the kubeadm join command from it. This includes dynamically created join tokens. With the local-exec provider, we create a shell script inside this project, then connect to the controller via SSH, grab the join command, and feed it into the shell script. Here is how:

resource "hcloud_server" "controller" {
  //...
  provisioner "local-exec" {
    command = <<EOF
      rm -rvf ./bin/03_kubeadm_join.sh
      echo "echo 1 > /proc/sys/net/ipv4/ip_forward" > ./bin/03_kubeadm_join.sh
      ssh root@${self.ipv4_address} -o StrictHostKeyChecking=no -i .ssh/id_rsa.key "kubeadm token create --print-join-command" >> ./bin/03_kubeadm_join.sh
    EOF
  }

Kubeadm Automation: Worker

The workers are provisioned in a very similar way: We copy and execute the installation script and the dynamically created kubeadm join shell script. Furthermore, since the workers need to wait for the controller to be fully provisioned, we add a depends_on relationship.

The complete configuration is this:

resource "hcloud_server" "worker" {
  //...
  depends_on = [
    hcloud_server.controller
  ]

  connection {
    type        = "ssh"
    user        = "root"
    private_key = tls_private_key.generic-ssh-key.private_key_openssh
    host        = self.ipv4_address
  }

  provisioner "remote-exec" {
    scripts = [
      "./bin/01_install.sh",
      "./bin/03_kubeadm_join.sh"
    ]
  }
}

Cluster Provisioning

All Terraform resources are ready. Now we can start the provisioning.

export TF_VAR_hcloud_token=SECRET

terraform apply

#...

Terraform will perform the following actions:

  # hcloud_server.controller["controller"] will be created
  + resource "hcloud_server" "controller" {

   # hcloud_server.worker["worker1"] will be created
  + resource "hcloud_server" "worker" {

#...
Do you want to perform these actions in workspace "staging"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

tls_private_key.generic-ssh-key: Creating...
tls_private_key.generic-ssh-key: Provisioning with 'local-exec'...
tls_private_key.generic-ssh-key (local-exec): (output suppressed due to sensitive value in config)
tls_private_key.generic-ssh-key: Creation complete after 2s [id=763061a58dc3b7c5b9c09b76e4ddbf4cdb52c3ab]
hcloud_ssh_key.primary-ssh-key: Creating...
hcloud_ssh_key.primary-ssh-key: Creation complete after 2s [id=8482290]
hcloud_server.controller["controller"]: Creating...
hcloud_server.controller["controller"]: Still creating... [10s elapsed]
hcloud_server.controller["controller"]: Provisioning with 'remote-exec'...
hcloud_server.controller["controller"] (remote-exec): Connecting to remote host via SSH...
#...

Run terraform show to see the controllers IP address, then grab the kubeconfig file via SSH:

ssh root@${CONTROLLER_IP} -o StrictHostKeyChecking=no -i .ssh/id_rsa.key "cat /root/.kube/config"

And with this, you can operate the cluster with kubectl:

kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
controller-staging   Ready    control-plane,master   22m     v1.23.11
worker1-staging      Ready    <none>                 49s     v1.23.11
worker2-staging      Ready    <none>                 5m51s   v1.23.11

Conclusion

This article showed how to use Terraform for provisioning a kubeadm cluster. We started with the configuration of the Terraform provisioners, then created a variable that holds the cluster configuration, and continued with the resource definition for the controller and worker. When this Terraform project runs, it creates the controller first, grabs its kubeadm join command, provisions the worker and joins them into a cluster. After about 2 minutes you have your full available cluster. Copy the kubeconfig to your machine and start working with your cluster.