The Kubernetes autoscaling tries to add more pods in horizontal scaling and more resources (like memory or CPU) are added to the pod in vertical scaling. but while you are deploying a bunch of dedicated memory stateful servers (like game servers) you need to scale the node sizes instead of adding more pod or increasing pod resources.

I developed an open-source configurable k8s scaler server for the stateful dedicated servers inspired by Mark Mandel's (founder of Agones.dev) blog post. He had an awesome presentation on GDC about developing, deploying, and scaling dedicated game servers.

Kubescaler

The Kubescaler is a simple Kubernetes node scaler for dedicated servers. It runs the scale function in a loop and checks all nodes’ resources to determine when to scale up and down.

Slot

The slot is a pod that runs the game server and is limited to certain resources.

Scaler

The scaler considers a buffer of slots that must be available for new pods, so at the start, it compares the available slots (that is equal to node resource capacity divided to slot’s needed resources), if the available slot is smaller than the buffer size, it tries to scale up the node pool size and in other condition, it scales down the node pool size.

Scheduling nodes

The Kubernetes allows marking a node as unschedulable. It shows that is the controller can schedule a pod on the node or not. The scaler uses this option to help the extra node get empty faster.

Scaling up

Before starting to resize the node pool size, the scaler checks unschedulable nodes and starts to mark them as schedulable until the needed resource be supplied. if no unschedulable nodes are found or the needed resource didn’t supply, it resizes the node pool size to the needed size.

Scaling down

If the available slot is greater than the buffer size, the scaler starts to mark extra nodes as unschedulable, this option prevents the controller to schedule a new pod on the extra node. at the end, it checks the unschedulable nodes (extra nodes) for running pods, if no dedicated server pod is running and the node empty time passed the expiration, it deletes the node from node pool. Note that adding a new node to the node pool takes time, so an expiration time is considered for the empty node before deleting them, maybe a little while later the needed resource increased.

Selectors

The scaler lists the scalable nodes using node-selector option and uses the pod label selector to determine that is the node is empty of dedicated server pods or not.

Podwatcher

The scaler server watches the dedicated server pod creation or deletion events and triggers the scale function in addition to the time interval.

Cloud Provider

To resize the node pool size or delete a node we need to use the cloud API. The Digitalocean cloud provider is implemented in the Kubescaler. feel free to contribute and implement any other cloud provider and send the PR.
To use the Digitalocean API, you must set the cloud-provider-token and the cluster and node pool names.

The docker image exists here, also feel free to contribute to the repo:

https://github.com/theredrad/kubescaler