I recently posted about the open-source release of our "always encrypted" K8s distro Constellation.

This post is about running Rocket.Chat on Constellation, which gives, well, "always encrypted" Rocket.Chat 🙂

This is cool, because "always encrypted" Rocket.Chat shields the entire server-side of Rocket.Chat from the (cloud) infrastructure and ensures that all data is always encrypted - even during processing in memory.

What is Constellation💫?

The idea of Constellation is to shield entire K8s clusters from the cloud infrastructure using Confidential VMs, which are available on recent AMD EPYC server CPUs. Constellation ensures that all data in the cluster is always encrypted (even during processing) and is inaccessible from the infrastructure. Based on the CPUs' remote attestation feature, Constellation makes this verifiable for the DevOps engineer.

What is Rocket.Chat🚀?

Rocket.Chat is an open-source communications platform. In less fancy terms, it's a chat server with a web frontend. Think of it as a self-hosted alternative to Slack. It's website touts it as a "secure and compliant communications platform".

Why do we need "always encrypted"🔒 Rocket.Chat🚀?

Rocket.Chat offers E2E encryption of messages but that comes with some downsides for UX (e.g., no more search and user needs to manage the key manually) and also, conceptually, cannot protect metadata, server configurations, code etc.

Consequently, different infrastructure-based attackers, for example someone with root access to a K8s node, can still learn and manipulate a lot. In fact, such an attacker could even inject malicious JS into client sessions and leak Rocket.Chat's E2E encryption keys or plaintext messages or just disable encryption. So it's probably fair to say that Rocket.Chat's E2E encryption mostly helps against passive attackers who are able to read or intercept some server-side data.

If we want to protect against the active attackers from the picture above, we need "always encrypted" Rocket.Chat 💪

Running Rocket.Chat🚀 on Constellation💫

Running Rocket.Chat on Constellation turned out to be pretty straightforward. In the first step, I created a Constellation cluster:

constellation create --control-plane-nodes 1 --worker-nodes 2 -y
constellation init
export KUBECONFIG="$PWD/constellation-admin.conf"

Note that I created my cluster on Azure. If you don't have an Azure or GCP subscription but want to try this out nonetheless, there is also MiniConstellation, which you can run locally and doesn't require confidential-computing hardware.

Installing Rocket.Chat

My goal was to make Rocket.Chat available under its own URL with TLS. For this, I installed and configured the following components via Helm charts on my freshly created Constellation cluster:

1. NGINX Ingress Controller
2. cert-manager for managing the TLS certificate
3. Constellation's azuredisk-csi-driver to get encrypted persistent storage on Azure (note that this will be automatically installed with the upcoming Constellation 2.2 release)
4. Finally, Rocket.Chat itself, which also brings MongoDB

If you're not in the mood to do this manually, we also created a combined Helm chart. The chart details how each component needs to be configured. After you set your desired hostname in the values.yaml, the chart can be installed as follows (assuming GoDaddy as domain provider):

export GODADDY_API_KEY=<your creds here>
export GODADDY_SECRET_KEY=<your creds here>
export MONGO_USER_SECRET=$(openssl rand -base64 32)
export MONGO_ROOT_SECRET=$(openssl rand -base64 32)
helm dependency update ./rocketchat
helm upgrade rocket-infra ./rocketchat --install --namespace default --set infra.enabled=true --set apiKey=$GODADDY_API_KEY --set secretKey=$GODADDY_SECRET_KEY
helm upgrade rocket-app ./rocketchat --install --namespace default --set app.enabled=true rocketchat.mongodb.auth.password=$MONGO_USER_SECRET --set rocketchat.mongodb.auth.rootPassword=$MONGO_ROOT_SECRET

Done! 🏃‍♂️🏁

The result is publicly accessible at https://rocket.edgeless.systems/ Feel free to come over and say "hi" 🙂

What does this give us?

Running Rocket.Chat on Constellation ensures that our entire deployment, including MongoDB and all other components that we installed above, is shielded from the infrastructure and that all data is always encrypted.

What does this mean?

Constellation ensures that all K8s nodes run on AMD-based Confidential VMs (CVMs). CVMs are strongly isolated from the host and remain encrypted in memory at runtime. Constellation also ensures that all nodes run the same minimal mkosi-based node image.

How does Constellation ensure these properties?

Constellation leverages the hardware-based "remote attestation" feature of CVMs. The Constellation CLI uses remote attestation to verify the first node in the cluster and to bootstrap a secure connection to the cluster. All subsequent nodes that are created in the cluster are verified by existing ones. During the verification process, it is checked if a "good" node image is running on "good" hardware with a "good" configuration. Constellation knows what a "good" node image is, because we sign and publish the measurements (i.e., hashes) of each node image we release.

Once a node has been verified, it gets access to a Cilium/Wireguard-based VPN, which is shared by the nodes. A verified node also receives a key for storage encryption. Nodes ensure that all data they write to the network or to storage is encrypted. The K8s admin doesn't have to deal with key management. It all happens automatically.

If we put all of this together, we get a K8s cluster that

1. keeps all data encrypted on the network, in storage, and in memory.
2. is isolated from the underlying infrastructure (incl. hypervisor and host OS).
3. is verified based on "remote attestation".

The Constellation docs have more details on the architecture and security features.

What about user connections?

Users connect to Rocket.Chat via the browser, e.g., by browsing to https://rocket.edgeless.systems/. TLS connections to Rocket.Chat on Constellation terminate inside CVMs, so the encryption is end-to-end.

But how does the user know for sure? Unfortunately, there is no direct way for the user to tell. However, they can manually inspect the TLS certificate (by clicking on the lock symbol in the browser bar) and confirm with their peers if its the expected certificate. The SHA-256 fingerprint of the server I created is CB 2C 2D 96 E6 B2 59 9D DC FD 99 C5 7A 80 58 60 3C A1 22 99 6D 28 BF FA AD 5B F0 52 EF F0 31 C5, btw. 🙂

Ideally, a browser plugin would exist that would allow pin a certificate for a domain. Ideally, the plugin would also be able to verify hardware-based remote attestation statements. This is future work 🙂 Even without such a plugin, Rocket.Chat on Constellation protects against many types of infrastructure-based attackers and accidental data leaks.

Wrap up

Hope you enjoyed this walkthrough. Personally, I'm super-excited by the possibilities that confidential computing and Constellation create. What do you think? Any other apps that you'd like to see become 🪄 "always encrypted"?

If you're interested in learning more about confidential computing, I recently wrote a whitepaper on the core concepts.

What are your thoughts about "always encrypted" Rocket.Chat, Constellation, or confidential computing in general?