Objectives

My objectives when setting up K3s in my home lab were to play around with Kubernetes and save on cloud costs by utilizing the computing resources to host CI runners for GitLab and GitHub. However, once complete, I decided it'd be fun and good practice to configure a deployment system linked to my K3S cluster as well.

This article is to possibly help anyone also interested in learning Kubernetes and setting up a CI/CD pipeline while saving on costs.

Here are the steps I took!

Preparation

Proxmox & K3s -

• I chose Proxmox as a hypervisor because it simplifies resource management and enables stable, scalable local Kubernetes environments.
• I'm using K3s because it offers Kubernetes features like scalability and security while using fewer resources and being quicker to deploy than K8s.

MetalLB -

• MetalLB is a solution to ensure services within a locally hosted Kubernetes cluster are accessible, load-balanced, and have their networking traffic adequately managed.

• Although MetalLB isn't recommended for production environments, it works well for this project as the objectives are focused on testing and development.

• For information on setting up MetalLB, take a look at their GitHub page here - https://github.com/metallb/metallb

TechnoTim has a great article & video covering how to automate the set-up and deployment of both K3s and MetalLB using Ansible: https://technotim.live/posts/k3s-etcd-ansible/

Steps

Example Code :
App CI Repo ~ https://github.com/swe-pea/ASP-NET-TestApp
Kubernetes Manifest/CD ~ https://github.com/swe-pea/home-lab/tree/main/ASP-TestApp

Setting Up The ASP.NET CI

• I started off by creating a GitHub repo dedicated to storing application code and building new containerized versions of the application.
• To self-host a single GitHub runner for the CI repository, I first set up an ACR (Actions Runner Controller) on K3s by applying the following manifest and registering a GitHub PAT as a Kubernetes secret

# Applies kubernetes config to set up ARC
kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml

kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_token=${GITHUB_TOKEN}

• Then, create and apply the following .yml file. By creating the secret and applying the following manifest, a single GitHub runner is created and linked to the CI repo.

# runner.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: example-runner
spec:
  repository: exampleUser/example-repo
  env: []

• Additionally, I created a DockerHub repository to store the application docker images built from the GitHub repository.

The GitHub workflows pushed with the application code automatically run on the newly configured self-hosted runner. Using docker/login-action@v3 and docker/build-push-action@v5, the containerized application is stored in DockerHub.

Deploying The ASP.NET Application

Having successfully set up the CI pipeline and containerized the ASP.NET application, it's time to deploy it to the K3S cluster.

1. Deployment Repository: I created a second GitHub repository with the sole purpose of deploying the ASP.NET application. This repository will use Kubernetes manifests (in .yml format) to define how the application should operate within the K3S cluster(replica count, ports, etc.)
2. ArgoCD: Using a deployment tool like ArgoCD makes it seamless to connect repositories to your Kubernetes cluster and deploy applications. Although I won't go into the details of how to set ArgoCD up, as a Kubernetes amateur, this is the method I went with for quickly deploying my Kubernetes manifest from the repository to my cluster.

• Helm Charts: Although Helm is a popular package manager tool for Kubernetes, with Helm charts offering pre-configured packages to make deployment quick and consistent, I decided to work with Kubernetes manifests directly to focus on the fundamentals.

• Continuous Deployment: Since the project objectives weren't focused on deployment strategies like rolling update, canary, or blue-green, continuous deployment was not configured. However, you can create a continuous deployment from this solution by using webhooks if you are interested.

Conclusion

In conclusion, this journey has provided insights into setting up a cost-effective, self-hosted CI/CD pipeline in a home lab, utilizing K3S and other OSS. I've learned a lot while also having lots of fun, so I'm hoping by sharing my experience working on this small project, someone else can maybe gain some inspiration or value from it as well!

Happy deploying,

-Ivan