TL;DR: We built our own reverse proxy using the Handshake packet allowing us to only turn servers on when someone wants to play hence reducing costs.

How did we get here

It all started when I managed to convince some friends to follow my idea for our final student project. Noticing that for every traditional game hosting platform you’re paying a flat monthly fee for a service constantly up whether you use it or not, essentially paying for an idle service most of the time. I wanted to provide a refreshing look at how gaming server hosting is done and bring the “pay only what you use” cloud philosophy to the gaming community.

We chose the popular game Minecraft, named our project Minekloud and oriented ourselves towards containerized workload hosted in a public Cloud provider’s managed Kubernetes. This alone might be worth its own article in the future but let’s focus on today’s subject. One of the many challenges we faced to create a plausible offer was for multiple servers to share the same IP thereby mutualising the Cloud load balancer and associated cost. This is not a brand new problem in our industry and the solution is to use some kind of reverse proxy (think of Traefik or NGINX). We called this component the Minecraft Gateway and added some interesting business logic such as starting the game server when a player tries to join or shut it down when no one is playing. In this blog post, I’ll share with you why we ended up building our own but most importantly how we did it.

The Minecraft protocol

First, let’s start with an overview of what happens when you try to connect to a Minecraft server. Your Minecraft client uses a special protocol on top of a TCP connection to talk with the server, it defines how information is exchanged between the two in the form of packets. Packets are just a sequence of bytes sent over the TCP connection. Everything from world data to entities movements is sent this way but here we’ll only focus on the first packets used to establish the connection.

There are 2 types of connections a Minecraft server can handle. The first one is the server status request, when you browse your multiplayer menu it’s sent to every server you added to retrieve their icon, MOTD, partial player list and ping latency. When a server doesn’t respond, it appears offline in your game client. The second one is when you join the server to play. For both cases, the first packet to be sent is the Handshake containing some connection information, the protocol version and the connection type (Status or Play).

The Play connection starts with a Handshake packet, followed by a Login Start packet containing the player username. Then they both exchange an Encryption Request and Response packet, it’s at this moment the server verifies that the client has authenticated itself with Mojang Studios and has a valid session. After this step the connection is secure and every following packet is encrypted using a shared secret.

I described the most common case but some server configuration options can have an impact. Particularly if the server is configured to run in offline mode, the server never verifies the user identity with Mojang Studios, it doesn’t send the Encryption Request and the connection is not encrypted.

Comparing existing solutions

The networking model we choose implied that every incoming traffic to the cluster must go through a Cloud load balancer which is not free and, similar to what ingress controllers do, we needed a way to mutualise them. Of course, we also looked at other networking modes with Kubernetes like direct access to nodes but having to deal with DNS TTL problems was a big no-no for us.

Using different ports

Before talking about reverse proxy we need to talk about another alternative, having multiple programs waiting for connections and running on the same computer and IP address is not uncommon. The most simple solution is to use a different port, each service listens on a different port number ranging from 0 to 65535. Most protocols define a default port number, for HTTP it’s 80 and Minecraft 25565. For Minecraft servers owners, using the default port is preferable as it means players don't have to specify the port number when filling in the server address in-game. We know we could have used the lesser-known Minecraft SRV record to hide the different ports from the players but either way, most Cloud providers limit the number of ports your load balancer can listen to so it wasn’t possible.

Traefik, NGINX, ...

In cloud projects, common tools used are Traefik, NGINX (and others) to handle HTTP requests (Layer 7) but as we’ve shown before, Minecraft uses its own protocol which is not HTTP. Those products also provide lower-level proxy capability at the TCP level (Layer 4), the problem is that at this layer you can’t distinguish which server players want to connect to. Although that would seem fun at first, users most certainly don’t want their players to join a random server we host each time they connect.

BungeeCord, Waterfall, ...

Now for the question you’ve probably been asking yourself since the beginning of this blog post if you’ve already worked with Minecraft servers before. Why don’t we just use BungeeCord or Waterfall? For those of you who don’t know, these solutions are usually used to connect multiple server instances together to support more players or multiple game modes. I won’t go into details but they act as an intermediary and allow you to switch players between servers without the need for them to reconnect. By design, they need to read every packet, therefore, terminating the connection encryption which forces the servers behind them to run in offline mode. It’s perfectly fine when owners use those tools to create their network but it will be very shady for a hosting provider to require every user to run their server in offline mode and trust it with security as it delegates player authentication to the proxy and essentially disables encryption.

Building our own

I’m still convinced user experience always comes first and they shouldn’t be bothered with constraints from the host’s underlying infrastructure, it should be transparent for them. The solution we found that worked best in our case was to create our own reverse proxy and only snoop the first unencrypted packets to route the player to the right destination.

Leveraging the Handshake packet

Packet composition

The handshake is composed of 4 fields of data. The first is the protocol version used by the client, it doesn’t follow the game version but is incremented every time a change is made and is currently 756 for Minecraft 1.17.1. The server address is the hostname or IP that was used by the client to connect, it’s the field you fill in when adding a new server in the multiplayer menu. There’s also the port number and the next state (always 1 for a status request and 2 for playing).

It’s the most important packet for our use case thanks to the Server Address field. It means we can have multiple hostname resolve to the same IP but still be able to differentiate them.

Routing players connections

We can leverage this Server Address field to route the player’s connection to the right server. Let’s for the sake of simplicity say we only got 2 servers (A and B), we assigned a subdomain for each of them, respectively foo.example.com and bar.example.com. It doesn’t even have to be a subdomain, it can be any hostname as long as its DNS record points to the Gateway. Its configuration is dynamic and the correspondence between hostname and server, stored in our API, is periodically fetched.

If we take a look at what happens step by step, the Client initiates the connection with our Gateway and sends a Handshake packet immediately followed by a Login Start packet as it does with every server. The Gateway reads these packets, extracts the server address field and finds the corresponding server. Once found, the Gateway initiates a connection with the server, sends those same packets it received prior. Its job is now done and it pipes the two connections together so that every data received from the client is sent to the server, and every data sent by the server is sent to the client. From the server perspective, there’s no visible difference and the Gateway doesn’t interfere with the Encryption so it can run in online mode.

Adding more features to the Gateway

Auto-start

Do you remember when we said we only want the server to be up and users to pay when someone actually wants to play? Well, that would be tedious for them to use a web interface and manually turn it on every time. Wouldn't it be nice if we could know when someone tries to join and automatically start the server for them? You might already see where this is going and you’re right, that’s possible and it’s exactly one of the first things we added to the Gateway. Please keep in mind that this feature was intended for our target audience of small servers between friends ( 4 to 5 players max) and doesn’t make as much sense for bigger communities when there’s always someone online.

When a player tries to join a turned off server, the Gateway informs the API which will automatically turn it on if the server owner enabled this feature. Unfortunately starting a Minecraft server is not an instantaneous process and will take at least 1-2 minutes. We can’t just leave the player on the loading screen for that long and it will probably timeout anyway. We just send him a carefully crafted Disconnect packet with a message informing him that the server is currently starting. We considered some solutions to get around the cold start but it’s out of scope for this article.

Servers status

Just like player connections, the Gateway is able to route status requests to the desired destination but when a server is not running, it can’t respond to status requests. In those situations, the gateway can respond on his behalf instead of forwarding the request. For example, if a server can be automatically started, we respond to the request so the server appears online and we add a custom MOTD stating that it can be started by the player.

Player count and auto-stop

It also keeps track of how many active connections exist for each server and shares this information with our API which will automatically stop an idle server after a defined amount of time. We read the player username in the Login Start packet and associate it with the connection. This is then used to feed the admin panel’s player list without requiring additional plugins to be installed or relying on the incomplete status response list.

Limitations

Although it was fine as a minimum viable product for a student project, it’s definitely not bulletproof in its current state. The main concern is abuse prevention, if someone hates you (this can happen sometimes on the internet) they could easily create a script to connect, start your server, and make sure it’s always running to cost you more money. Another limitation is that from the server perspective, every user comes from the same place (the gateway) and shares the same IP address. If you were to ban a player by IP, you would essentially ban everyone using this gateway instance (even possibly yourself). A solution for some compatible server distributions would be to support IP forwarding or the proxy protocol v2. Bukkit and Spigot have configuration options to do this with BungeeCord or Waterfall but it’s hard to find more information without digging through the source code.

Conclusion

Thanks for reading this far, it was a really fun and interesting project to work on! The reverse proxy was just a small part of the whole Minekloud project, there are so many more interesting things to share that I might do another article if you’re interested. The source code for the Gateway and every other component of our project is free and open-source (MIT License), you can check the link down below.

Useful links

Project source code (MIT license)
GitLab - minekloud/minecraft-gateway

Similar project
GitHub - itzg/mc-router - Shout-out to itzg for all his projects in the containerized Minecraft community!

Great Minecraft communities for developers
r/admincraft
r/syscraft

Minecraft protocol
MCDevs - wiki.vg

The proxy protocol
Specifications