TheLazy.Dev
Portfolio GitHub

Blog

kubernetes |devops |opensource |cloud

Kubernetes volumes upside-down with Discoblocks

mhmxs

Richard Kovacs

Posted on May 17, 2022

Kubernetes volumes upside-down with Discoblocks

In the stone age, almost everybody had the same answer regarding running stateful applications on top of Kubernetes: "Don't do it dude at all!". The main reasons behind this philosophy are persistent storage in the eco-system did its first baby steps at that time, and no one had experience with long-running storage systems in Kubernetes. As a storage space engineer, I can tell you that data loss is not a joke. The storage system is one of the most critical environments and needs lots of care before we say: stable. Just remember BTRFS was in "testing" for almost a decade. But I think it is time to swap the paradigm (if you haven't done it yet) and start using Kubernetes with its full power. Why do I say that? Because there are nice options on the market to create replicated persistent volumes or permanent and reusable cloud disks. Ondat.io is one of the solutions, for more info, I suggest watching this nice video. But Ondat isn't the topic of this article :)

Please let me introduce our new open-source declarative disk configuration system: Discoblocks. It is in an early alpha state at the moment, but we put significant effort into turning it's production-ready soon. So USE IT YOUR OWN RISK, please!

What are Discoblocks good for?
Discoblocks is a disk and volume manager for Kubernetes helping to automate CRUD (Create, Read, Update, Delete) operations for cloud disk device resources attached to Kubernetes cluster nodes.

Why use Discoblocks?
Discoblocks can be leveraged by cloud-native data management platform to manage the backend disks in the cloud.

When using such a data management platform to overcome the block disk device limitation from hyperscalers, a new set of manual operational tasks needs to be considered.

At the current stage, Discoblocks is leveraging the available hyperscaler CSI (Container Storage Interface) within the Kubernetes cluster to:

  • introduce a CRD (Custom Resource Definition) per workload with
    • capacity
    • mount path within the Pod
    • nodeSelector
    • podSelector
    • upscale policy
  • provision the relevant disk device using the CSI when the workload deployment will happen
  • monitor the volume(s)
  • resize automatically the volume based on the upscale policy

Currently, the only supported cloud driver is AWS EBS, but more coming. CSI driver has a really simple interface in Discoblocks:

// Driver public functionality of a driver
type Driver interface {
        // IsStorageClassValid validates StorageClass
        IsStorageClassValid(*storagev1.StorageClass) error
        // GetPVCStub creates a PersistentVolumeClaim for driver
        GetPVCStub(string, string, string) (*corev1.PersistentVolumeClaim, error)
        // GetCSIDriverPodLabels returns the labels of CSI driver Pod
        GetCSIDriverPodLabels() (string, map[string]string)
}
Enter fullscreen mode Exit fullscreen mode

Can you tell me a real-world use case?
Imagine you have a website and you would like to store media files on a managed persistent volume. On the regular way, you have to create a PersistentVolumeClaim, then you have to open the manifests of the service, seek for workload part, and edit the workload at two places to attach PVC. Not a big deal, but requires some manual work, and the question is how many times do you have to do it? If the answer is exactly 1, the rest of this article is not for you, otherwise please continue reading :)

How does it work in action?
The first step is to create the cluster:

: ${NAME?= required}
: ${AWS_ACCOUNT_ID?= required}

# Create EKS cluster with EBS
eksctl create cluster --version 1.21 --name $NAME
Enter fullscreen mode Exit fullscreen mode

Enable EBS addon on the cluster:

eksctl utils associate-iam-oidc-provider --cluster $NAME --approve
issuer=$(aws eks describe-cluster --name $NAME --query 'cluster.identity.oidc.issuer' --output text | sed 's|https://||')
now=$(date +'%s')
cat > aws-ebs-csi-driver-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$issuer"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$issuer:aud": "sts.amazonaws.com",
          "$issuer:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
        }
      }
    }
  ]
}
EOF
aws iam create-role --role-name $NAME-EKS_EBS_CSI_DriverRole-$now --assume-role-policy-document file://'aws-ebs-csi-driver-trust-policy.json'
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --role-name $NAME-EKS_EBS_CSI_DriverRole-$now
aws eks create-addon --cluster-name $NAME --addon-name aws-ebs-csi-driver --service-account-role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/$NAME-EKS_EBS_CSI_DriverRole-$now
Enter fullscreen mode Exit fullscreen mode

Create StorageClass:

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF
Enter fullscreen mode Exit fullscreen mode

And finally, install Cert Manager:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
Enter fullscreen mode Exit fullscreen mode

The cluster is up and running, it is time for the magic:

kubectl apply -f https://github.com/ondat/discoblocks/releases/download/v0.0.3/discoblocks_v0.0.3.yaml
Enter fullscreen mode Exit fullscreen mode

Please make sure Discoblocks is up and running:

kubectl get deploy -n kube-system discoblocks-controller-manager
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
discoblocks-controller-manager   1/1     1            1           37s
Enter fullscreen mode Exit fullscreen mode

Create a disk configuration custom resource:

cat <<EOF | kubectl apply -f -
apiVersion: discoblocks.ondat.io/v1
kind: DiskConfig
metadata:
  name: diskconfig-nginx
  namespace: default
spec:
  # Name of the underlaying StorageClass
  storageClassName: ebs-sc
  # Initial capacity of the disk
  capacity: 1Gi
  # Where to mount the disk
  mountPointPattern: /usr/share/nginx/html/media-%d
  # What kind of workloads requires the volume
  podSelector:
    app: nginx
  # Resize policies
  policy:
    upscaleTriggerPercentage: 50
    maximumCapacityOfDisk: 2Gi
EOF
Enter fullscreen mode Exit fullscreen mode

As the title mentions Discoblocks turns volume management upside-down. That means you don't have to change your manifests to attach a volume to the workload, the tool does the magic in the background. So use kubectl command to create a deployment:

kubectl create deployment --image=nginx nginx
Enter fullscreen mode Exit fullscreen mode

Check the result:

kubectl describe po -l app=nginx
Name:         nginx-6799fc88d8-dkjtm
Namespace:    default
Containers:
  nginx:
    Mounts:
      /usr/share/nginx/html/media-0 from discoblocks-2064555022-2155331895-2470140894 (rw)
  discoblocks-metrics:
    Image:         bitnami/node-exporter:1.3.1
    Port:          9100/TCP
    Command:
      /opt/bitnami/node-exporter/bin/node_exporter
      --collector.disable-defaults
      --collector.filesystem
Enter fullscreen mode Exit fullscreen mode

It seems Discoblocks has created the disk, attached it to the workload, and added a sidecar for measuring volumes.

Well done mate, but how does automatic resize work?
Here is the current state of the underlying PVC:

kubectl get pvc discoblocks-2064555022-2155331895-2470140894
NAME                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
discoblocks-2064555022-2155331895-2470140894   Bound    pvc-636102fd-be8e-4861-88d2-63f51474f427   1Gi        RWO            ebs-sc         7m11s
Enter fullscreen mode Exit fullscreen mode

So generate some data to the volume:

kubectl exec nginx-6799fc88d8-dkjtm -c nginx -- dd if=/dev/zero of=/usr/share/nginx/html/media-0/data count=1000000
Enter fullscreen mode Exit fullscreen mode

Check mount point:

kubectl exec nginx-6799fc88d8-dkjtm -c nginx -- df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    976M  491M  470M  52% /usr/share/nginx/html/media-0
Enter fullscreen mode Exit fullscreen mode

As you see more than 50% of the disk is used, Discoblocks should resize the volume:

kubectl get pvc discoblocks-2064555022-2155331895-2470140894
NAME                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
discoblocks-2064555022-2155331895-2470140894   Bound    pvc-636102fd-be8e-4861-88d2-63f51474f427   2Gi        RWO            ebs-sc         10m
Enter fullscreen mode Exit fullscreen mode

It did, how it looks like in the application side:

kubectl exec nginx-6799fc88d8-dkjtm -c nginx -- df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    2.0G  492M  1.5G  25% /usr/share/nginx/html/media-0
Enter fullscreen mode Exit fullscreen mode

Hurray!!!

Please feel free to give it a ride. Don't hesitate to share your experiences and thoughts with us. And finally: enjoy the new world of volume configuration!

💖 💪 🙅 🚩
mhmxs
Richard Kovacs

Posted on May 17, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

👋 Why we say GOODBYE to HELM 👋
kubernetes 👋 Why we say GOODBYE to HELM 👋

December 11, 2023

Using Helm dashboard and intents-based access control for pain-free network segmentation
kubernetes Using Helm dashboard and intents-based access control for pain-free network segmentation

August 9, 2023

Comparing Kubernetes Gateway and Ingress APIs
kubernetes Comparing Kubernetes Gateway and Ingress APIs

February 1, 2023

Karpenter: Tool for Autoscaling Kubernetes Clusters
kubernetes Karpenter: Tool for Autoscaling Kubernetes Clusters

December 29, 2022

Kubernetes volumes upside-down with Discoblocks - #2
kubernetes Kubernetes volumes upside-down with Discoblocks - #2

September 23, 2022