DHH just coined the term "Citadel," which finally gives us an excellent way to reference how we approach tech at AppSignal. We said, "Hey, this is us! Our thing has a name now".

In addition to the Majestic Monolith, someone should write up the pattern of The Citadel: A single Majestic Monolith captures the majority mass of the app, with a few auxiliary outpost apps for highly specialized and divergent needs.

— DHH (@dhh) April 7, 2020

To explain how AppSignal uses the Citadel pattern, we'll share a bit on how our system works. AppSignal is a monitoring product that has a user-facing application and an API that the monitoring agent sends data to. This data is then processed and turned into graphs and insights.

Monolith

The application our customers interact with is a monolithic Rails app, with parts of the front-end written in React. The backend is entirely written in Ruby and talks to a few databases (we split out data from different customers to separate clusters for scaling reasons). This Rails app also handles a bunch of other tasks such as sending out alerts to external services.

When we started this Rails app processed incoming data from our monitoring agent as well, we foresaw that data ingestion would turn out to be a bottleneck. So we used a Sinatra app running on a subdomain that ingested data and created Sidekiq jobs that were processed by the Rails app.

Growing Pains

This architecture worked well for years. As our business grew, it became clear that the specific task of processing incoming data from the agents was going to need special treatment. When you're monitoring billions and billions of requests, you run into hard limits. The main limiting factor wasn't so much that Ruby is slow (we all know that it is not 😉), but that the way we had architected things caused too much locking in our databases.

An Outpost

We looked at several possibilities and then decided Kafka was the best fit for our situation. We had some experience with Rust and thought that its speed and reliability would be a very good fit for this system. We rewrote our data ingestion and processing system in Rust, using Kafka as a combination of queue and storage system.

We only moved the incoming data processing part of the Rails app to this outpost service. The rest of the systems works well in the form of a monolithic app. We understand it deeply, and we like keeping things simple. The monolith still handles most of the logic and interacts with Kafka heavily. Our wish to keep our monolith led to us writing a Kafka gem, so the main app can communicate with the outpost easily.

If you want to learn more about how Kafka works at AppSignal check out the Railsconf talk I gave about this.

Life at the Citadel

This brings us to the current situation where we are very happy in our citadel. As DHH said:

A single Majestic Monolith captures the majority mass of the app with a few auxiliary outpost apps for highly specialized and divergent needs.

In our case, we have a single outpost service for our highly specialized needs. If there were a RailsConf this year, we would have given DHH some extra stroopwafels as appreciation for giving it a name. 🍪