The Fully Encapsulated Monitoring Notebook

Imagine, in a page of code, hosted entirely on the web, with zero tools to install or infra to manage, you could have a state-of-art programmable data visualization system recording anything reachable on the internet. Bold claim? I have built two such systems and I am so excited by the possibilities I wish to excitedly share the architecture.

IDE, Documentation, Code Host and Front End: Observablehq.com

Observablehq.com is a mind-blowingly simpler platform that can replace: Github, CI, VScode, static hosting, Google docs, and containers. It hosts, runs, and provides an IDE for client-side Javascript. Furthermore, it provides a novel reactive notebook format, so code can be written non-linearly and the program output is interleaved with the code. It’s also a literate programming environment, so notebooks also support documentation, multiplayer commenting, and collaboration.

Observable is a single tool that can replace numerous critical tools required in a software project. Including development, technical documentation, and hosting.

Charting: @observablehq/plot

Observablehq.com, is the hottest place for Dataviz, so it naturally hosts a number of state-of-the-art tools such as plot. Plot stands out by being so fast to develop with, but Vega-lite is another great option. Check out some of the workshops on how to use these tools effectively.

(Plot) Analyzing Time Series Data in Observable: Hands-on Workshop
(vega-lite) Observable: Vega-Lite: A Crash Course

To do queries on the client side I would use Arquero for data operations. There is an interactive intro to Arquero through Data Wrangler.

UI: @observablehq/inputs

For simple controls, Observable has an excellent selection of off-the-shelf components through @observablehq/inputs. With some work, you can make complex UIs too (see Scaling User Interface Development).

Realtime Database: firebase.google.com

As Observable is a reactive client-side environment, it works really well with reactive client-side databases like Firebase. When I am choosing for monitoring systems, I look for: cheap writes and low latency. So I tend to pick the Firebase Realtime Database over Firestore for monitoring applications. (FWIW I pick Firestore for online transaction processing applications).

API Endpoints: WEBCode.run

Monitoring solutions often need APIs, so they can respond to external services (e.g. queries, datapoint ingestion, OAuth handshakes). WEBCode.run extends Observable with an SDK that allows notebooks to host serverless endpoints. These endpoints can utilize secrets and scheduled functions so are a great fit when you need to interface with external systems and it can be all done without leaving observablehq.com.

Example Monitoring Notebooks

I have built two notebooks so far that are self-enclosed systems, and they have both provided immense value to me. I intend to do many more.

1. Realtime Inbound Request Log

The Realtime Inbound Request Log notebook hosts an endpoint and puts all inbound requests in a log. When you view the notebook the log entries are queried and some summaries are provided. This notebook was developed to help me sniff request headers on a remote machine not under my control, by scripting the remote machine to query the instrumented endpoint.

2. Latency Monitor

The Latency Monitor notebook is a full end-to-end prober solution. It periodically schedules traffic to a system, measures the latency, and summarizes the results in a dashboard.

This has been in operation for nearly a1 year and helped quantify latency performance.

Everything in one end-to-end environment.

I am delighted that everything is in a single artifact that can be shared with a URL. This is not some low code compromise either, all the parts work together providing the end-user with a state-of-the-art experience. Realtime data dashboards using state-of-the-art DataViz libraries, stitched together with a modern language using tools that support version control, step debuggers, etc.

By having the tools to modify the monitoring pipeline next to the technical documentation and realtime output of the production data allows anyone to maintain it with no context switching.

Overall it is a joyful development experience and operationally simple.