Kubernetes Cluster Management and Cloud Automation

Projectsveltos, Crossplane, and ClusterAPI are three open source projects that can be used together to simplify the management of Kubernetes clusters.

ClusterAPI is a Kubernetes subproject that provides declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. ClusterAPI runs in a management cluster and can be used to manage clusters that are hosted on a variety of infrastructure providers, including on-premises, public clouds, and edge computing platforms.
Projectsveltos is a Kubernetes add-on controller that makes it easy to deploy and manage add-ons and applications in Kubernetes clusters. Add-ons and applications that need to be deployed by Projectsveltos can be expressed as templates. Projectsveltos will instantiate these templates before deploying them in the matching managed clusters. The values used to instantiate the templates can be fetched by Projectsveltos at runtime from resources in the management Kubernetes cluster. This allows Projectsveltos to dynamically customize the deployment of add-ons and applications based on the specific needs of each managed cluster.
Crossplane is an open source Kubernetes-native control plane that facilitates the management of cloud infrastructure and services across various cloud providers and on-premises environments. Crossplane is designed to be declarative and API-driven, making it easy to manage infrastructure using familiar Kubernetes tools and workflows.

In this article, we will show how to use Projectsveltos, Crossplane, and ClusterAPI to create a Google Cloud Storage bucket and a simple application that uploads a file to the bucket. By the end of this article, you will have a basic understanding of how to use Projectsveltos, Crossplane, and ClusterAPI to manage Kubernetes clusters.

Installation

ClusterAPI, Projectsveltos, and Crossplane are all deployed in the management cluster, which is a Kubernetes cluster that manages other Kubernetes clusters. From the management cluster, programmatically, other Kubernetes clusters can be created and add-ons and applications can be deployed.

To create a management cluster with Projectsveltos, ClusterAPI and Docker as infrastructure provider:

git clone https://github.com/projectsveltos/addon-controller
make quickstart

Alternatively, you can follow instructions on ClusterAPI documentation and Projectsveltos documentation.

In this tutorial we have a managed cluster powered by ClusterAPI with Docker as infrastructure provider.

kubectl get clusters -A
NAMESPACE   NAME                  PHASE         AGE     VERSION
default     clusterapi-workload   Provisioned   4m50s   v1.27.0

Following are the instructions to deploy Crossplane in the management cluster:

# Authenticate with Google Cloud
gcloud auth login

# Generate a key for a service account
gcloud iam service-accounts keys create service-account-key.json --iam-account=[SA-EMAIL]

# Export the GCP project-id you want storage bucket to be created on
export GCP_PROJECT_ID=<project-id>

helm repo add crossplane-stable https://charts.crossplane.io/stable

helm repo update

# Following sets up Crossplane 
helm install crossplane crossplane-stable/crossplane \
--namespace crossplane-system --create-namespace

# Create a Kubernetes secret with the GCP credentials 
$ kubectl create secret generic gcp-secret \
-n crossplane-system  --from-file=creds=<PATH TO YOUR GCP Credentials.json>

# Install the crossplane GCP provider 
cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-storage
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-storage:v0.35.0
EOF

# Create a ProviderConfig
cat <<EOF | kubectl apply -f -
apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: $GCP_PROJECT_ID
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-secret
      key: creds
EOF

Verify the provider installed with kubectl get providers

kubectl get providers
NAME                          INSTALLED   HEALTHY   PACKAGE                                                AGE
provider-gcp-storage          True        True      xpkg.upbound.io/upbound/provider-gcp-storage:v0.35.0   8m56s
upbound-provider-family-gcp   True        True      xpkg.upbound.io/upbound/provider-family-gcp:v0.36.0    8m50s

Create a Bucket for each managed cluster

In this tutorial, we want Sveltos to coordinate with Crossplane to create a Google Storage Bucket for each managed cluster and then deploy an application in each managed cluster that uploads a file to the proper bucket.

This requires solving the following problems:

The operations must be run in a precise order. For each matching cluster, first Projectsveltos must instruct Crossplane to create a Google Storage Bucket. Only after Crossplane is done creating the bucket, Projectsveltos can proceed further deploying a Pod in the managed cluster that will upload a file to the bucket.
The Pod must be passed the URL of the Google Storage Bucket to use. Projectsveltos will fetch this information from the management cluster (the Crossplane Bucket instances) and will use it to instantiate a template representing a Pod that will upload a file to the storage bucket.

Let's build the Projectsveltos ClusterProfile YAML to achieve all that.

Crossplane creating Google Storage Bucket

A Bucket instance is a Kubernetes custom resource (CR) that instructs Crossplane to create a Google Cloud Storage bucket. Crossplane will then update the status section of the CR with information about the bucket, such as its name, location, and URL.

apiVersion: storage.gcp.upbound.io/v1beta1
kind: Bucket
metadata:
  name: crossplane-bucket-default-clusterapi-workload
  ...
spec:
  forProvider:
    location: US
    project: projectsveltos
    publicAccessPrevention: inherited
    storageClass: STANDARD
  ...
status:
  atProvider:
    ...
    id: crossplane-bucket-default-clusterapi-workload
    ...
    url: gs://crossplane-bucket-default-clusterapi-workload

To instruct Projectsveltos to create a Bucket instance[1]:

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
spec:
  ...
  policyRefs:
  - deploymentType: Local
    kind: ConfigMap
    name: bucket
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:   
  name: bucket
  namespace: default
  annotations:
    projectsveltos.io/template: "true"
data:       
  bucket.yaml: |
    apiVersion: storage.gcp.upbound.io/v1beta1
    kind: Bucket
    metadata:
     name: crossplane-bucket-{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}
     labels:
       docs.crossplane.io/example: provider-gcp
       clustername: {{ .Cluster.metadata.name }}
       clusternamespace: {{ .Cluster.metadata.namespace }}
    spec:
      forProvider:
        location: US
      providerConfigRef:
        name: default

Above is instructing Projectsveltos to:

Fetch the ConfigMap named bucket in the namespace default.
Take the content of the ConfigMap (the data section).
This content is a template for a Crossplane Bucket instance. Projectsveltos will instantiate the template before deploying it (the projectsveltos.io/template annotation is set).
The template refers to the namespace and name of the ClusterAPI-powered cluster. This ensures that Projectsveltos creates a single Crossplane Bucket instance for each matching cluster.
After the template is instantiated, Projectsveltos will deploy it to the management cluster (since the ClusterProfile.PolicyRefsdeploymentType is Local).

Once Projectsveltos creates a Bucket instance in the management cluster, Crossplane takes over and creates a Google Cloud Storage bucket. Projectsveltos then pauses and waits for Crossplane to finish creating the bucket.

Deploy Pod to managed cluster

When Crossplane finishes creating a Google Cloud Storage bucket, we need Projectsveltos to resume and deploy a Pod in the managed cluster. This Pod needs information about the bucket, such as its name and URL. Crossplane has added this information to the status section of the Bucket CR.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  templateResourceRefs:
  - resource:
      apiVersion: storage.gcp.upbound.io/v1beta1
      kind: Bucket
      name: crossplane-bucket-{{ .ClusterNamespace }}-{{ .ClusterName }}
    identifier: CrossplaneBucket

The YAML section that instructs Projectsveltos to fetch the Bucket CR is as follows:

In the templateResourceRefs section, add a resource reference to the Bucket CR. The resource reference should specify the following:

The API version and kind of the Bucket CR.
The name of the Bucket CR, which is a template that depends on the cluster namespace and name.
An identifier for the resource reference. In this example, we are using the identifier CrossplaneBucket.

Projectsveltos will fetch all of the management cluster resources listed in the templateResourceRefs section before trying to deploy any resources to the managed cluster. In this specific example, we are telling Projectsveltos to fetch the Bucket CR instance, whose name is a template that depends on the cluster namespace and name.

Sveltos will react to changes to any resources listed in the templateResourceRefs section. Once the Bucket CR instance is updated by Crossplane, Sveltos will fetch it and then deploy the Pod in the managed cluster.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  ...
  policyRefs:
  ...
  - deploymentType: Remote
    kind: ConfigMap
    name: uploader
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: uploader
  namespace: default
  annotations:
    projectsveltos.io/template: "true"
data:
  ...
  pod.yaml: |
    apiVersion: v1
    kind: Pod
    metadata:
      name: create-and-upload-to-gcs
      namespace: default
      annotations:
        bucket: {{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}
    spec:
      containers:
      - name: uploader
        image: google/cloud-sdk:slim
        command: ["bash"]
        args:
          - "-c"
          - |
            echo "Hello world" > /tmp/hello.txt
            gcloud auth activate-service-account --key-file=/var/run/secrets/cloud.google.com/service-account.json
            gsutil cp /tmp/hello.txt gs://{{ (index .MgtmResources "CrossplaneBucket").metadata.name }}
        volumeMounts:
          - name: gcp-sa
            mountPath: /var/run/secrets/cloud.google.com/
            readOnly: true
      volumes:
        - name: gcp-sa
          secret:
            secretName: gcs-credentials

Above is the YAML section which instructs Sveltos to deploy Pod in the managed cluster:

Add a reference to the uploader ConfigMap in the policyRefs section. The deploymentType is set to Remote because we want Sveltos to deploy the content of the ConfigMap in the managed cluster.
The ConfigMap data section contains a Pod expressed as a template (the projectsveltos.io/template annotation is set). This template references information from the Bucket CR that Sveltos fetched in the management cluster.
Sveltos will instantiate this template and deploy it to managed cluster.

Complete YAML

Following YAML is all that is required. A ClusterProfile and two ConfigMap instances.

Since the Pod also needs credentials to write to bucket, Sveltos is also deploying a Secret with GCP credentials that Pod will mount and use.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector: env=fv
  templateResourceRefs:
  - resource:
      apiVersion: storage.gcp.upbound.io/v1beta1
      kind: Bucket
      name: crossplane-bucket-{{ .ClusterNamespace }}-{{ .ClusterName }}
    identifier: CrossplaneBucket
  - resource:
      apiVersion: v1
      kind: Secret
      namespace: crossplane-system
      name: gcp-secret
    identifier: Credentials
  policyRefs:
  - deploymentType: Local
    kind: ConfigMap
    name: bucket
    namespace: default
  - deploymentType: Remote
    kind: ConfigMap
    name: uploader
    namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:   
  name: bucket
  namespace: default
  annotations:
    projectsveltos.io/template: "true"
data:       
  bucket.yaml: |
    apiVersion: storage.gcp.upbound.io/v1beta1
    kind: Bucket
    metadata:
     name: crossplane-bucket-{{ .Cluster.metadata.namespace }}-{{ .Cluster.metadata.name }}
     labels:
       docs.crossplane.io/example: provider-gcp
       clustername: {{ .Cluster.metadata.name }}
       clusternamespace: {{ .Cluster.metadata.namespace }}
    spec:
      forProvider:
        location: US
      providerConfigRef:
        name: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: uploader
  namespace: default
  annotations:
    projectsveltos.io/template: "true"
data:
  secret.yaml: |
    apiVersion: v1
    kind: Secret
    metadata:
      name: gcs-credentials
      namespace: default
      annotations:
        bucket: "{{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}"
    type: Opaque
    data:
      service-account.json: {{ $data:=(index .MgtmResources "Credentials").data }} {{ (index $data "creds") }}
  pod.yaml: |
    apiVersion: v1
    kind: Pod
    metadata:
      name: create-and-upload-to-gcs
      namespace: default
      annotations:
        bucket: {{ (index .MgtmResources "CrossplaneBucket").status.atProvider.url }}
    spec:
      containers:
      - name: uploader
        image: google/cloud-sdk:slim
        command: ["bash"]
        args:
          - "-c"
          - |
            echo "Hello world" > /tmp/hello.txt
            gcloud auth activate-service-account --key-file=/var/run/secrets/cloud.google.com/service-account.json
            gsutil cp /tmp/hello.txt gs://{{ (index .MgtmResources "CrossplaneBucket").metadata.name }}
        volumeMounts:
          - name: gcp-sa
            mountPath: /var/run/secrets/cloud.google.com/
            readOnly: true
      volumes:
        - name: gcp-sa
          secret:
            secretName: gcs-credentials

We created a ClusterProfile that instructs Sveltos to create a Bucket CR in the management cluster.
We told Sveltos to wait for Crossplane to create a Google Cloud Storage Bucket, fetch the Bucket CR and then deploy a Pod in the managed cluster.
The Pod will upload a file to the Bucket CR using the GCP credentials that Sveltos deployed in the managed cluster.

Summary

ClusterAPI provides a consistent way to manage Kubernetes clusters across different infrastructure providers. This makes it easy to deploy and manage Kubernetes clusters in a variety of environments.
Projectsveltos provides a way to manage Kubernetes add-ons and applications using GitOps principles. This makes it easy to automate the deployment and management of Kubernetes add-ons, and to track changes to the cluster configuration.
Crossplane provides a way to manage infrastructure resources using Kubernetes Custom Resource Definitions (CRDs). This makes it easy to manage infrastructure resources in a consistent way, and to automate the provisioning and management of infrastructure resources.

By using ClusterAPI, Projectsveltos, and Crossplane together, you can get the following benefits:

Consistency: You can manage Kubernetes clusters and infrastructure resources in a consistent way, regardless of the underlying infrastructure provider.
Automation: You can automate the deployment and management of Kubernetes clusters and infrastructure resources, which can save time and effort.
Track-ability: You can track changes to the cluster configuration and infrastructure resources, which can help you to troubleshoot problems and to audit changes.
Reliability: You can improve the reliability of your Kubernetes clusters and infrastructure resources by automating the provisioning and management of these resources.

👏 Support this project

If you enjoyed this article, please check out the Sveltos GitHub repo and star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with Sveltos. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.

If you have any feedback, bugs, or PRs, please feel free to contribute to the project. Your contributions will help make Sveltos even better.

And finally, please help spread the word about Sveltos by starring the repo or sharing it with your friends and colleagues. Thank you for reading!

[¹]: Sveltos needs to be granted permission to manage lifecycle of Crossplane Bucket instance. Add following RBACs.

kubectl edit clusterrole addon-controller-role-extra

- apiGroups:
  - storage.gcp.upbound.io
  resources:
  - buckets
  verbs:
  - get
  - list
  - watch
  - patch
  - delete
  - create

Blog