Kubernetes with Amazon Elastic Kubernetes Service

There are several options to roll your own Kubernetes cluster. In addition to provisioning on managed infrastructure, you can also create a managed cluster that is integrated with a particular cloud environment, enabling you to use other services and concepts from the cloud environment.

This article is a hands-on tutorial to setup the Kubernetes platform Elastic Kubernetes Service (EKS) on AWS. Step-by-step, you will learn how to download the required binaries, get the AWS authorization, draft a EKS configuration, and the cluster installation procedure. You will also see how to access the cluster with both kubectl and SSH and learn how to perform a cluster update.

This article originally appeared at my blog admantium.com.

Part 1: Prerequisites

To follow this tutorial, you need to have a fully registered and activated AWS account. This account also needs to have complete access rights for managing theses required resources: AWS instances, VPC, IAM user.

You also need a dedicated computer to install the required binaries and perform the installation steps, and you will also connect from this machine to the cluster, either by using kubectl or SSH. This machine is called the EKS controller in the remainder of this article.

Finally, resource creation and utilization generate costs - be sure you are willing to pay for this experience! In a three-day period, I paid $30. You can keep a good overview about all costs with the cost dashboard - but only after they incurred.

Part 2: Tool Installation

The first task is to install all the necessary binary tools on the EKS controller.

kubectl

The binary for interacting with a Kubernetes cluster. It can be installed from your package manager, or you can just grab the binary suitable for your architecture.

To install version 1.24.0 On a Linux host, execute these commands:

cd ~
curl -LO https://dl.k8s.io/release/v1.24.0/bin/linux/amd64/kubectl
chmod +x kubectl
mkdir -p ~/.local/bin
mv ./kubectl ~/.local/bin/kubectl

eksctl

This CLI tool creates and manages the Kubernetes cluster. It’s provided by Amazon as a binary file. Go to its Github releases page and grab the binary for your OS.

For a Linux host, execute these commands:

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
chmod +x /tmp/eksctl
mkdir -p ~/.local/bin
mv /tmp/eksctl ~/.local/bin/eksctl

Part 3: Cluster Configuration

The cluster creation is a one-time command. The cluster itself is either preconfigured, or can be customized by flags or a config file. Configuration options are extensive - see the full configuration file documentation for all options. In this tutorial, I'm following the preconfigured path, using mostly default settings.

To get started, we need to define the following things:

Name: The name of the cluster
Region: The AWS availability region in which you want to deploy the cluster
Kubernetes version: Check the supported EKS versions and ensure that you have a compatible kubectl version installed
Node number: How many worker nodes the cluster should have
AWS instance (optionally): The instance type of the nodes, see the AWS documentation. If you do not provide a setting, the default instance type m5.large? will be used.
Node type: In EKS, K8S Workloads can run either as container on the ECS instances, which means container in VMs, or as containers on bare-metal server with Amazon Fargate.

In addition to these, you need to provide your AWS credentials, for example as the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Part 4: Cluster Installation

First Try: Console

My first try to install a cluster was to pass all options as flags. This was the command:

eksctl create cluster \
  --region=eu-central-1 \
  --version=1.23 \
  --nodes=2 \
  --fargate

Log output:

2022-11-05 12:37:51 [ℹ]  eksctl version 0.117.0
2022-11-05 12:37:51 [ℹ]  using region eu-central-1
2022-11-05 12:37:51 [ℹ]  setting availability zones to [eu-central-1a eu-central-1b eu-central-1c]
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1a - public:192.168.0.0/19 private:192.168.96.0/19
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1b - public:192.168.32.0/19 private:192.168.128.0/19
2022-11-05 12:37:51 [ℹ]  subnets for eu-central-1c - public:192.168.64.0/19 private:192.168.160.0/19
2022-11-05 12:37:51 [ℹ]  nodegroup "ng-83b22df4" will use "" [AmazonLinux2/1.23]
2022-11-05 12:37:51 [ℹ]  using Kubernetes version 1.23
2022-11-05 12:37:51 [ℹ]  creating EKS cluster "scrumptious-hideout-1667648262" in "eu-central-1" region with managed nodes
2022-11-05 12:37:51 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2022-11-05 12:37:51 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=eu-central-1 --cluster=scrumptious-hideout-1667648262'
2022-11-05 12:37:51 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "scrumptious-hideout-1667648262" in "eu-central-1"
2022-11-05 12:37:51 [ℹ]  CloudWatch logging will not be enabled for cluster "scrumptious-hideout-1667648262" in "eu-central-1"
2022-11-05 12:37:51 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=eu-central-1 --cluster=scrumptious-hideout-1667648262'
2022-11-05 12:37:51 [ℹ]
2 sequential tasks: { create cluster control plane "scrumptious-hideout-1667648262",
    2 sequential sub-tasks: {
        wait for control plane to become ready,
        create managed nodegroup "ng-83b22df4",
    }
}
2022-11-05 12:37:51 [ℹ]  building cluster stack "eksctl-scrumptious-hideout-1667648262-cluster"
2022-11-05 12:37:52 [ℹ]  deploying stack "eksctl-scrumptious-hideout-1667648262-cluster"

My first attempt was not successful. The eksctl command was stuck with the deploying stack message as shown above. So, I ran the command again, but gave a dedicated name to the cluster:

eksctl create cluster \
  --name=staging \
  --region=eu-central-1 \
  --version=1.23 \
  --nodes=2 \
  --fargate

This resulted in the same message. And running it again showed an error that the cluster is available, but I could not access it. What’s happing here? Logging into the AWS management console, I saw that several security groups, VPS, network interfaces and elastic IPs of the cluster name were created, but no instances. Hmm.

The deletion of all these resources was a cumbersome, manual process, which can only be achieved in this order: First, detach all network interfaces, then delete the elastic IPS, then delete the network interfaces, and finally delete the VPCs.

Second Try: Config File

The second try was to use a config file. You can either use a bare-bones configuration file, e.g. from the eksctl.io documentation, or create one with the command eksctl --dry-run. I used the latter and then customized the resulting file.

But take care! Running just eksctl create cluster --dry-run creates a very different file then a more customized eksctl create cluster --name=staging --region=eu-central-1 --dry-run. Use the first, bare command. For completions sake, here is the full config file with which the installation was successful.

# eksctl create cluster --name=staging --dry-run

apiVersion: eksctl.io/v1alpha5
availabilityZones:
- eu-central-1c
- eu-central-1b
- eu-central-1a
cloudWatch:
  clusterLogging: {}
iam:
  vpcResourceControllerPolicy: true
  withOIDC: false
kind: ClusterConfig
kubernetesNetworkConfig:
  ipFamily: IPv4
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  disableIMDSv1: false
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false
      appMesh: false
      appMeshPreview: false
      autoScaler: false
      awsLoadBalancerController: false
      certManager: false
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: false
      fsx: false
      imageBuilder: false
      xRay: false
  instanceSelector: {}
  labels:
    alpha.eksctl.io/cluster-name: staging
    alpha.eksctl.io/nodegroup-name: ng-c3cdc337
  maxSize: 2
  minSize: 2
  name: ng-c3cdc337
  privateNetworking: false
  releaseVersion: ""
  securityGroups:
    withLocal: null
    withShared: null
  ssh:
    allow: false
    publicKeyPath: ""
  tags:
    alpha.eksctl.io/nodegroup-name: ng-c3cdc337
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 80
  volumeThroughput: 125
  volumeType: gp3
metadata:
  name: staging
  region: eu-central-1
  version: "1.23"
privateCluster:
  enabled: false
  skipEndpointCreation: false
vpc:
  autoAllocateIPv6: false
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: false
    publicAccess: true
  manageSharedNodeSecurityGroupRules: true
  nat:
    gateway: Single

In this file, I merely changed the cluster name to eksctl-superb-hedgehog and modified managedNodeGroups.ssh to obtain SSH access to the nodes:

managedNodeGroups:
  - amiFamily: AmazonLinux2
    #...
    ssh:
      allow: true
      publicKeyPath: "~/.ssh/eksctl_rsa.key"

The cluster creation took a long time - searching the internet about eksctl installations hanging, I found this stackoverflow thread: cluster creation can take up to 20 minutes!

And indeed, it took about this time:

2022-11-07 20:08:36 [ℹ]  eksctl version 0.117.0
2022-11-07 20:08:36 [ℹ]  using region eu-central-1
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1b - public:192.168.0.0/19 private:192.168.96.0/19
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1a - public:192.168.32.0/19 private:192.168.128.0/19
2022-11-07 20:08:36 [ℹ]  subnets for eu-central-1c - public:192.168.64.0/19 private:192.168.160.0/19
2022-11-07 20:08:36 [ℹ]  nodegroup "ng-9bf41c12" will use "" [AmazonLinux2/1.23]
2022-11-07 20:08:36 [ℹ]  using Kubernetes version 1.23
2022-11-07 20:08:36 [ℹ]  creating EKS cluster "superb-hedgehog-staging" in "eu-central-1" region with managed nodes
2022-11-07 20:08:36 [ℹ]  1 nodegroup (ng-9bf41c12) was included (based on the include/exclude rules)
2022-11-07 20:08:36 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2022-11-07 20:08:36 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
# ....
2022-11-07 20:08:37 [ℹ]  deploying stack "eksctl-superb-hedgehog-staging-cluster"
2022-11-07 20:09:07 [ℹ]  waiting for CloudFormation stack "eksctl-superb-hedgehog-staging-cluster"
# ...
2022-11-07 20:26:49 [ℹ]  waiting for CloudFormation stack "eksctl-superb-hedgehog-staging-nodegroup-ng-9bf41c12"
2022-11-07 20:26:49 [ℹ]  waiting for the control plane to become ready
2022-11-07 20:26:51 [✔]  saved kubeconfig as "~.kube/config"
# ...
2022-11-07 20:26:53 [✔]  EKS cluster "superb-hedgehog-staging" in "eu-central-1" region is ready

Looking into the AWS management console, I could see the created instances and volumes:

Part 5: Cluster Access and Maintenance

With the installation completed, a custom kubeconfig file is automatically placed at ~/.kube/config. Use it to gain cluster access:

export KUBECONFIG=~/.kube/config

kubectl  get nodes

NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-34-211.eu-central-1.compute.internal   Ready    <none>   12m   v1.23.9-eks-ba74326
ip-192-168-73-218.eu-central-1.compute.internal   Ready    <none>   11m   v1.23.9-eks-ba7432

For the SSH access, you need to get the nodes' public DNS name from the AWS management console, and then connect as the ec2-user.

ssh ec2-user@ec2-3-70-174-13.eu-central-1.compute.amazonaws.com
Last login: Sat Nov  5 02:57:35 2022 from 205.251.233.105

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
12 package(s) needed for security, out of 18 available
Run "sudo yum update" to apply all updates.

[ec2-user@ip-192-168-69-138 ~]$ kubelet --version
Kubernetes v1.23.9-eks-ba74326

Part 6: Cluster Updates

For checking how cluster updates work, I completely reinstalled a new cluster and used Kubernetes version v1.22.

As before, the cluster creation took some time to complete.

2022-11-09 20:11:11 [ℹ]  eksctl version 0.117.0
2022-11-09 20:11:11 [ℹ]  using region eu-central-1
2022-11-09 20:11:11 [ℹ]  subnets for eu-central-1b - public:192.168.0.0/19 private:192.168.96.0/19
#...
22-11-09 20:28:59 [ℹ]  node "ip-192-168-69-138.eu-central-1.compute.internal" is ready
2022-11-09 20:29:01 [ℹ]  kubectl command should work with "/Users/guenthers/.kube/config", try 'kubectl get nodes'
2022-11-09 20:29:01 [✔]  EKS cluster "staging" in "eu-central-1" region is ready

The upgrade happens gradually. You need to choose the new Kubernetes version, and then pass it to upgrade commands. The upgrade commands targets are: control plane node, kube-proxy, aws-nodes, and finally coredns.

Here are the concrete update commands and their results:

eksctl upgrade cluster --name=staging --version=1.23
2022-11-09 20:47:44 [ℹ]  re-building cluster stack "eksctl-staging-cluster"
2022-11-09 20:47:44 [✔]  all resources in cluster stack "eksctl-staging-cluster" are up-to-date
2022-11-09 20:47:44 [ℹ]  checking security group configuration for all nodegroups
2022-11-09 20:47:44 [ℹ]  all nodegroups have up-to-date cloudformation templates

eksctl utils update-kube-proxy --cluster=staging --approve
2022-11-09 20:48:53 [ℹ]  "kube-proxy" is now up-to-date

eksctl utils update-aws-node --cluster=staging --approve
2022-11-09 20:49:21 [ℹ]  skipped existing "kube-system:ServiceAccount/aws-node"
2022-11-09 20:49:21 [ℹ]  replaced "CustomResourceDefinition.apiextensions.k8s.io/eniconfigs.crd.k8s.amazonaws.com"
2022-11-09 20:49:21 [ℹ]  replaced "ClusterRole.rbac.authorization.k8s.io/aws-node"
2022-11-09 20:49:22 [ℹ]  replaced "ClusterRoleBinding.rbac.authorization.k8s.io/aws-node"
2022-11-09 20:49:22 [ℹ]  replaced "kube-system:DaemonSet.apps/aws-node"
2022-11-09 20:49:22 [ℹ]  "aws-node" is now up-to-date

eksctl utils update-coredns --cluster=staging --approve
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:Service/kube-dns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:ServiceAccount/coredns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:ConfigMap/coredns"
2022-11-09 20:49:47 [ℹ]  replaced "kube-system:Deployment.apps/coredns"
2022-11-09 20:49:48 [ℹ]  replaced "ClusterRole.rbac.authorization.k8s.io/system:coredns"
2022-11-09 20:49:48 [ℹ]  replaced "ClusterRoleBinding.rbac.authorization.k8s.io/system:coredns"
2022-11-09 20:49:48 [ℹ]  "coredns" is now up-to-date

I encountered no error, all deployments/daemonsets are running. To check the results, I connected via SSH to the nodes took the kubectl version. Also, you can see that image tags of certain deployments or daemonsets reflect the new Kubernetes version:

kubectl describe daemonset kube-proxy -n kube-system
Name:           kube-proxy
#...
  Containers:
   kube-proxy:
    Image:      602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/kube-proxy:v1.23.13-eksbuild.1

Part 7: EKS Cluster Internals

I was curious to find out which Kubernetes components the cluster uses. Running kubectl get all -A revealed these facts:

Linux distribution: amazon-eks-ami
Container runtime: containerd
Network communication: coredns, kube-proxy, amazon-vpc-cni
Ingress: aws-load-balancer-controller

Conclusion

Amazon EKS is a managed Kubernetes distribution. In this article, you learned all the essentials to get you started: a) Ensure to have a fully registered AWS account, b) install the required binaries eksctl and kubectl, c) create the cluster by issuing a dry-run installation, then customize the resulting configuration file. Cluster installation can take up to 20 minutes - don’t stop this in the middle, or you will need to manually cleanup several AWS resources. Once completed, you can connect via kubectl and SSH to the nodes. Finally, you also learned the required steps for upgrading the Kubernetes version of the cluster, and saw some cluster internals.

Blog