Cloud native homelab, Part 1: Hardware

santisbon

santisbon

Posted on September 10, 2023

Cloud native homelab, Part 1: Hardware

In this first installment of the series on building a cloud native homelab, we go over selecting your hardware.

The emphasis here is on cost optimization so we're not getting a cluster board, a bunch of compute modules (general purpose or machine learning), servers, a rack mount, or anything like that. We'll start out with one or two single-board computers (SBCs) that we can set up easily without a big capital expense. This will allow us to:

  • Deploy new nodes to our homelab in an automated fashion, just like on a public cloud with cloud-init.
  • Set up a Kubernetes cluster and deploy workloads to it using cloud native technologies for container orchestration, storage, and more.
  • Interact with the real world through sensors, cameras, motors, and other hardware if we choose to.

The most popular SBC is Raspberry Pi but finding one during a shortage may be hard. Fortunately at the time of this writing the shortage is over and you can now easily find one at a reasonable price. There are cheaper options, though, and you might be tempted to buy something like the Orange Pi. Let's go over the differences to show exactly what you're getting with each.

TL;DR Get the Raspberry Pi if you want a cloud native homelab

Image description

Raspberry Pi 4B Orange Pi 3B
Origin United Kingdom China
RAM 8GB LPDDR4-3200 SDRAM 8GB LPDDR4/4x
Storage Micro-SD card slot Micro SD slot.
Supports an eMMC module: 16GB - 256GB.
SPI Flash: 16MB/32MB.
M.2 M-KEY slot for SATA3 or PCIe 2.0 NVMe SSD.
Price (USD) $75 + shipping $45 + shipping
OS Raspberry Pi OS, Ubuntu, RetroPie, Recalbox, OctoPi, Home Assistant, others Orange Pi OS [1], Ubuntu, Debian, Android, others
Official images from Ubuntu and Debian? Yes No. Custom Ubuntu and Debian images provided by Orange Pi.
cloud-init included? Yes (at least on Ubuntu) No
Default Ubuntu package repo ubuntu.com huaweicloud.com
Linux kernel release [4] 5.15.0-1037-raspi (kernel extra modules available) 5.10.160-rockchip-rk356x (kernel extra modules not available)
CPU Broadcom BCM2711 Quad core Cortex-A72 (ARM v8) 64-bit SoC 1.8GHz Rockchip RK3566 Quad-core Cortex-A55 (ARM v8) 64-bit SoC 1.8GHz
Graphics Processing Unit (GPU) Integrated ARM Mali G52 2EE (Bitfrost) [2]
Neural Processing Unit (NPU) - Rockchip NPU AI accelerator with 0.8Tops@INT8 performance.
RKNN model type supports conversion of Caffe / TensorFlow / TFLite / ONNX / PyTorch / Keras / Darknet models
Vision Processing Unit (VPU) - Yes
Graphics & Video OpenGL ES 3.1, Vulkan 1.0.
H.265 (4K 60fps decode).
H.264 (1080p 60fps decode, 1080p 30fps encode).
GPU
OpenGL ES 3.2/2.0/1.1, Vulkan 1.1, OpenCL 2.0.
Embedded high performance 2D acceleration hardware.

VPU
H.265/H.264/VP9 (4K 60fps decode).
H.265 (1080p 100fps encode).
H.264 (1080p 60fps encode).
Ethernet Gigabit Gigabit
Wireless Wi-Fi 5 (802.11ac)
Bluetooth 5.0, BLE
Wi-Fi 5 (802.11ac)
Bluetooth 5.0, BLE
Wired antenna (not shown). Useful when building into a metal housing.
USB 2 USB 3.0 ports
2 USB 2.0 ports
1 USB 3.0 HOST
2 USB 2.0 HOST
1 USB 2.0 Device or HOST mode
General Purpose Input/Output (GPIO) 40 pin GPIO header 40 pin header supporting interface types: GPIO, UART, I2C, SPI, PWM
Video output 2 × micro-HDMI up to 4K 60fps.
1 × 2-lane MIPI DSI display port.
1 × HDMI TX 2.0 up to 4K 60fps.
1 × 2-lane MIPI DSI display port.
eDP1.3
Camera 1 × 2-lane MIPI CSI camera port 1 × 2-lane MIPI CSI camera port
Audio 4-pole stereo audio and composite video port 3.5mm headphone jack audio in/out
Power 5V DC via USB-C (minimum 3A) [3].
5V DC via GPIO header (minimum 3A) [3]
Power over Ethernet (PoE) enabled (requires separate PoE HAT).
5V DC via USB-C (3A)
Button - 1 MaskROM key
1 RESET key
1 POWER key
Fan Use the GPIO pins (3.3V or 5V) 2.54mm pin pitch 2-Pin 1.25mm pin pitch 5V fan connector
Real-time clock (RTC) Use the GPIO pins (3.3V) 2.54mm pin pitch 2-Pin 1.25mm pin pitch battery backup connector
Dimensions 85mm x 56mm x 11mm
46g
85mm x 56mm x 17mm
49g
[1] Orange Pi OS doesn't support the GPU, NPU, or VPU. But Debian, Ubuntu, and Android support them.
[2] arm.com, Rockchip GitHub
[3] A good quality 2.5A power supply can be used if downstream USB peripherals consume less than 500mA in total.
[4] Linux kernel extra modules nvme_fabrics and nvme_tcp are required for cloud native storage like OpenEBS.
  • You'll also need a 5V 3A power supply for each board if you're not setting them up with Power over Ethernet (PoE).
  • You don't need a mouse and keyboard for the boards because you'll manage them over SSH from your laptop. Even if you installed a graphical desktop, you could still access it remotely with a VNC viewer.
  • If you're not setting them up with Wi-Fi, you'll need an ethernet cable (CAT 5e or better) for each board.
  • For many applications you can use the Pi by itself without heat sinks or a fan and monitor the temperature in millidegree Celsius with cat /sys/class/thermal/thermal_zone*/temp. But we're going to be running some resource-intensive software like Kubernetes. For our cloud native homelab I recommend getting heat sinks, a fan, and a case to house everything from the start. The Raspberry Pi ARM cores and GPU will start throttling when they reach around 80°C.
  • Speaking of cases, I recommend getting a cluster case. It's way cheaper than a cluster board and it houses your Pis in a stackable way so it's great for expanding your homelab in the future. This one from the UK is particularly nice and made with Perspex (a high-quality acrylic) but there are other options on Amazon with faster shipping.
  • For storing large amounts of data you'll need external storage like an SSD drive. I recommend a SATA SSD although if you already have an NVMe SSD you can use that as well, just know that your board won't take full advantage of its speeds. This one is a great option along with this cable. Just make sure the cable/adapter has an ASMedia chipset.

With Raspberry Pi it's easy to automate the setup with cloud-init. You're also able to have cloud native replicated storage with OpenEBS thanks to the extra kernel modules for NVMe over Fabrics. If you want your homelab to be cloud native, Raspberry Pi is the way to go.

For Orange Pi, cloud-init is not included. On top of that, the images provided by Orange Pi can't be mounted on macOS so you can't inject the user-data anyway (and you have to use the images they provide, others won't work). That means you'll have to set it up manually. There's also the issue of not being able to use cloud native storage due to a lack of extra modules for that kernel release. Having said that, Orange Pi's AI capabilities might be a good addition to your homelab if you need a low-cost AI-accelerated IoT device and don't need any cloud native features like an automated setup or replicated storage.

If you have a 32-bit SBC like the Raspberry Pi 2, keep in mind that it can't run some Kubernetes distributions like MicroK8s so whatever SBCs you choose make sure they're 64-bit. If you have a Pi 2 it's great as a dedicated retro gaming console with Recalbox.

In the next part of this series, we'll set up our nodes as a Kubernetes cluster.

💖 💪 🙅 🚩
santisbon
santisbon

Posted on September 10, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Cloud native homelab, Part 1: Hardware
raspberrypi Cloud native homelab, Part 1: Hardware

September 10, 2023