Hello folks,

Have you ever wondered how do cloud providers build their cloud ? How do circle-ci, github actions, gitlab-ci do run your tests and builds your projects in their cloud ?

Don't wonder anymore, because in this series, i'll show to build your own mini-cloud at home.

Disclaimer:

• it will require some hardware investment,
• it will give you a glimpse of hardware architecture, but not full picture (we will not cover spine/leaf nodes architecture used in cloud datacenters)

What hardware do you need ?

• At least two raspberry pi with 4GB each (official price 55$ each),
• One MicroSD card for each Raspberry pi (i recommend taking at least a 32GB card),
• [optional] A Raspberry pi POE+ hat for each Raspberry pi (official price 20$ each)
• [optional] A POE+ ethernet switch (or the official raspberry pi power supply if you can't afford a POE switch and want to use wifi for networking)
• [Optional] A cluster case for a clean setup.

I made one based on 4 Raspberry pi's with POE+ hat and a POE+ switch. Total Budget : 400$.

If you go with only two RPI's over WIFI, it will cost you around 150$.

What it looks like?

night mode

day mode

Install Ubuntu server ARM64 on SD card

We could install the official Raspberry pi OS on each card, but unfortunately, raspberry only support 32 bit images officially (their 64 bit OS is still broken from time to time), and we want to use the full power of Raspberry pi 4 which is 64 bit. See this benchmark to compare for yourself the huge difference :

So here i'll recommend using Ubuntu Server 20.04 LTS ARM64 build provided by Raspberry pi Imager.

Select Other general purpose OS>Ubuntu> Ubuntu server 20.04 LTS

And click on the Write button.

SD Card Modification

Before putting your SD card on your raspberry pi, you first need to make a few changes to allow you to use it in Headless mode (ie: without a screen and keyboard linked to it).

With your SD card still inserted, you should see two new disks :

• bootfs
• writable

Enable SSH

Still at the root of bootfs disk, create an empty file named ssh :

touch ssh

This will enable ssh connections on your Raspberry pi.

Enable container cpu and memory isolation

Open a Terminal and go to the bootfs filesystem of your SD Card, and open the cmdline.txt file.
You should see a single line, append this at the end of the line :

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

This will enable you to use kubernetes (k3s to be precise) on your cluster.

Optional Enable POE+ hat driver

Open userconfig.txt and append these lines at the end of the file :

dtoverlay=rpi-poe
dtparam=poe_fan_temp0=45000
dtparam=poe_fan_temp1=55000
dtparam=poe_fan_temp2=65000
dtparam=poe_fan_temp3=75000

This will enable POE+ hat driver and fan configuration to cool down your Raspberry pi.

Rename your Rasberrys

Now go to the writable disk and edit as an admin the etc/hosts and append at the end:

127.0.1.1 <your-new-name>

For the first SD card, i suggest naming it to recognize that's it will be the master node of your cluster. Mine is called k3s-main. For the other ones, you can choose a predictable name like k3s-worker-01 ...

Also edit etc/hostname and change the current name (ubuntu) to the name you just choose.

Insert your SD card and boot

Now you can insert your SD card in your Raspberry pi's. put power on and let them boot.

Connect to each of them with ssh

Now you need to connect to each of your booted Raspberry pi's. If you have wifi and a dhcp server, you might be able to connect to them using their name with .local appended to it. For exemple :

> ssh ubuntu@k3s-main.local

When prompted for password, just use ubuntu. it will ask you to change the default password after a successfull login.
Once the password changed, it will disconnect you.

If it does not work, don't worry. First run ifconfig to get information about your network.

> ifconfig
etho0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet <YOUR.LOCAL.IP>  netmask 255.255.0.0  broadcast 172.19.255.255
        ether <MAC:ADDRESS>  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This should give you your local IP. replace the last two digits by 00 and run nmap :

> nmap -sn <YOUR.LOCAL.IP.ZERO>/24

This should return your your IP, plus the one of your Raspberry pi's. you should now be able to login using ssh and the raspberry IP

> ssh ubuntu@<Rasbpberry.IP>

If you have already setup a ssh key, i recommand using it to automatically connect to your Raspberry pi's without password prompt. To do this :

> ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@k3s-main.local

Enter password for the last time when prompted.

Update your Raspberry pi's

On each Raspberry pi's within the SSH session, run :

> sudo apt update & sudo apt upgrade

Install k3s

Main node

On your main node (k3s-main for me), run the following command :

> curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Once the script finished execution, you should be able to see your node running :

> kubectl get nodes
NAME      STATUS    ROLES     AGE     VERSION
k3s-main   Ready    control-plane,master    1m      v1.21.0

Save your cluster token

save your token to allow your workers to join the main node

> cat /var/lib/rancher/k3s/server/node-token
<YOUR-TOKEN>

Join main node on workers

On each of your worker nodes (k3s-worker-01, k3s-worker-02, k3s-worker-03 for me) join the main node by running the following command :

> curl -sfL https://get.k3s.io | K3S_URL=https://k3s-main.local:6443 K3S_TOKEN=<YOUR-TOKEN> sh -

Once the scripts finished execution, you should be able to see your nodes running (execute on main node):

> kubectl get nodes
NAME      STATUS    ROLES     AGE     VERSION
k3s-main   Ready    control-plane,master    15m      v1.21.0
k3s-worker-01   Ready     <none>    3m      v1.21.0
k3s-worker-02   Ready     <none>    2m      v1.21.0
k3s-worker-03   Ready     <none>    1m      v1.21.0

Conclusion

So we have installed a cluster on raspberry pi's. what now ?

Next time, we'll see how to install rancher on your cluster to monitor and administrate it with a beautifull UI.