Build own Kubernetes - Node setup
Jonatan Ezron
Posted on October 11, 2022
In the last posts we handled pods, Now in this article, we are going to focus on Node.
But first, what is a node?
Node can be a physical or virtual machine. There is a master node which contains the control plane and etcd and worker nodes which contains the running pods, kubelet, and k-proxy. In this article, we will focus on the worker node creation.
Before this article I had some refactor the code, The creating and running methods of the pods, output set to log file and not STDIO, added a new method to get the pod logs, and the create pod commands now calls a NewPodAndRun function which creates and runs the code.
I was struggling to find the right platform for the node os, at first I looked for an appropriate docker container image that will run the agent (kubelet) and containerd service in the background but with no luck. So we need to build some base image on our own, this node VM can also be created with KVM, I decided docker for the easier approach.
We first need to build a Dockerfile with Golang, we will use the base image ubuntu (I have tried alpine but had some troubles building the go source code 😔 ), install go and containerd, to check if everything works I have taken a simple HTTP server:
package main
import (
"fmt"
"net/http"
"log"
"os"
"os/exec"
)
func hello(w http.ResponseWriter, req *http.Request) {
fmt.Fprintf(w, "hello\n")
}
func headers(w http.ResponseWriter, req *http.Request) {
for name, headers := range req.Header {
for _, h := range headers {
fmt.Fprintf(w, "%v: %v\n", name, h)
}
}
}
func startContainerd() {
cmd := exec.Command("containerd")
cmd.Stdout = os.Stdout
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
log.Printf("just ran subprocess %d", cmd.Process.Pid)
}
func main() {
startContainerd()
http.HandleFunc("/hello", hello)
http.HandleFunc("/headers", headers)
http.ListenAndServe(":8090", nil)
}
The server first starts a containerd service process and then starts the server.
For the Dockerfile we installed Go, containerd, and copy and build or main.go:
FROM ubuntu
WORKDIR /agent
RUN apt-get update \
&& apt-get install -y wget git gcc \
&& wget -P /tmp https://go.dev/dl/go1.19.2.linux-amd64.tar.gz \
&& tar -C /usr/local -xzf "/tmp/go1.19.2.linux-amd64.tar.gz"
ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
COPY main.go .
RUN go build -o main main.go
RUN apt-get install -y containerd
EXPOSE 8090
ENTRYPOINT [ "./main" ]
We build and run with enlarged CPU and memory limit (the sizes for the CPU and memory I have got from minikube implementation) so we have enough for the pods we will create inside:
❯ sudo docker build . -t containerd_test
❯ sudo docker run -it --memory="2900MB" --cpus="2" -p 8090:8090 containerd_test
❯ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e26fd47de621 containerd_test "./main" 25 seconds ago Up 24 seconds 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp reverent_solomon
❯ sudo docker exec e26 ctr c ls
CONTAINER IMAGE RUNTIME
As you can see the HTTP server is running and there is a containerd process running as well.
Now we need to build our agent, the base functionality of creating pods is already set, we just need some API endpoints.
We will use the echo framework for the REST API, to implement the different API endpoints: POST for creating pods, DELETE for deleting, GET for getting the pods, and for getting pod logs.
Let's start with the main function of the agent binary, it will be in pkg/agent/agent.go that will start the containerd process and config the echo REST API:
package main
import (
"fmt"
"log"
"os"
"os/exec"
"github.com/jonatan5524/own-kubernetes/pkg/agent/api"
"github.com/labstack/echo/v4"
"github.com/labstack/echo/v4/middleware"
)
func initRoutes(e *echo.Echo) {
e.POST("/pod", api.CreatePod)
e.GET("/pod/:id/log", api.LogPod)
e.GET("/pod", api.GetAllPods)
e.DELETE("/pod/:id", api.DeletePod)
}
func initMiddlewares(e *echo.Echo) {
e.Use(middleware.LoggerWithConfig(middleware.LoggerConfig{
Format: "method=${method}, uri=${uri}, status=${status}\n",
}))
e.HTTPErrorHandler = func(err error, c echo.Context) {
c.Logger().Error(err)
e.DefaultHTTPErrorHandler(err, c)
}
}
func startContainerd() {
cmd := exec.Command("containerd")
cmd.Stdout = os.Stdout
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
log.Printf("containerd run on %d", cmd.Process.Pid)
}
func main() {
startContainerd()
e := echo.New()
initMiddlewares(e)
initRoutes(e)
e.Logger.Fatal(e.Start(fmt.Sprintf(":%s", api.PORT)))
}
We used port 10250 for the agent API because this is the same port used in the Kubernetes agent.
Next is the pkg/agent/api/pod.go that will contains the different functions that will handle the pod endpoint:
package api
import (
"net/http"
"github.com/jonatan5524/own-kubernetes/pkg/pod"
"github.com/labstack/echo/v4"
)
type podDTO struct {
ImageRegistry string `json:"image registry"`
Name string `json:"name"`
}
func CreatePod(c echo.Context) error {
podDto := new(podDTO)
if err := c.Bind(podDto); err != nil {
return err
}
id, err := pod.NewPodAndRun(podDto.ImageRegistry, podDto.Name)
if err != nil {
return err
}
return c.JSON(http.StatusCreated, podDTO{
ImageRegistry: podDto.ImageRegistry,
Name: id,
})
}
func LogPod(c echo.Context) error {
logs, err := pod.LogPod(c.Param("id"))
if err != nil {
return err
}
return c.String(http.StatusOK, logs)
}
func GetAllPods(c echo.Context) error {
pods, err := pod.ListRunningPods()
if err != nil {
return err
}
return c.JSON(http.StatusCreated, pods)
}
func DeletePod(c echo.Context) error {
if _, err := pod.KillPod(c.Param("id")); err != nil {
return err
}
return c.NoContent(http.StatusNoContent)
}
And as mentioned above, we are creating a node image of the node container, the Dockerfile will be located in the project root so we can copy all the project folders:
# Dockerfile for node image
FROM ubuntu
WORKDIR /agent
RUN apt-get update \
&& apt-get install -y wget git gcc \
&& wget -P /tmp https://go.dev/dl/go1.19.2.linux-amd64.tar.gz \
&& tar -C /usr/local -xzf "/tmp/go1.19.2.linux-amd64.tar.gz"
ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
RUN apt-get install -y containerd
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN go build -o main pkg/agent/agent.go
EXPOSE 8090
ENTRYPOINT [ "./main" ]
Let's build and test our work!
We build and run our container node with the previous requirements::
❯ sudo docker build -t containerd_test .
❯ sudo docker run -it --memory="2900MB" --cpus="2" -p 10250:10250 --privileged --name test --rm containerd_test
And now if we send a request to create a Redis pod:
❯ curl -X POST localhost:10250/pod -H 'Content-Type: application/json' -d '{"name": "redis", "image registry": "docker.io/library/redis:alpine"}'
{"image registry":"docker.io/library/redis:alpine","name":"redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5"}
The pod is created!
In the agent logs:
2022/10/10 15:36:07 pod created: redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5
2022/10/10 15:36:07 starting po
We can test and see that also all the other endpoints works:
❯ curl localhost:10250/pod
["redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5"]
❯ curl localhost:10250/pod/redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5/log
1:C 10 Oct 2022 15:36:07.757 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 10 Oct 2022 15:36:07.757 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
...
❯ curl -X DELETE localhost:10250/pod/redis-c5ad79db-0e8e-4526-a9d1-366cefd0e9d5
Everything works!
On the next article we will focus on automating the node creation and other methods on node with commands like the pods.
The full source code can be found here, the changes were in pkg/agent and Dockerfile.
Posted on October 11, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
