Three examples of abstraction and generalisation in systems architecture

There are many motivations that drive people to achieve greater and better things. Software developers have very specific motivations. The main one is the satisfaction of seeing a tedious manual process made easier for the workers who'll use it, and the knowledge that if we do well we'll get that gratification. The need to feel a sense of achievement at the end of every day and every project is also a powerful and seductive motivator.

But there are other, more abstract pleasures. One of these is the joy of abstraction itself  -  the distillation of common processes into a generalised form that can be put to work in many scenarios. Abstraction is a way of thinking that can create elegance out of complexity. And a desire for perfection and beauty in systems can lead to a never-ending abstraction-fuelled journey with multiple destinations and many achievements.

Many Splendoured Forms

Once upon a time, back in the late 1990s, I was starting out developing a new system for our latest client. I started where we always start, with the database, and then the forms to create, read, update or delete data  -  or "CRUD" as it's called in techie circles. And then an overwhelming sense of boredom and tedium came over me. I had just done the same for another client. The same but different. There was no way I could take what I done for the other customer and use it for this customer  -  the feature requirements were simply too different, and the data requirements were about as different as they could get. But the task was the same.

There is a kind of ultimate laziness which compels some people to work tirelessly and zealously to ensure that they will never have to do the same work ever again. That's what gripped me, and not for the first time in my working life. So I started to think about how we might generalise this process of creating data entry forms. This meant making it data-driven. The idea was to be able to create a relational database and then rapidly generate standards-based data-entry forms at the touch of a button.

I went through several possible technological approaches. I wrote some sample code for a few techniques to see which would be most performant and not drive us down any blind alleys. I showed my business partner Joel. He liked what I'd done. Thus, for the first time in this process I got confirmation that my thinking was largely correct, or at least not completely insane. And this peer-review of an idea is always the first go/no go point in many speculative projects.

"There is a kind of ultimate laziness which compels some people to work tirelessly and zealously to ensure that they will never have to do the same work ever again"

By the end of the summer that year, in around 10,000 lines of code, we had Velociraptor (V-Raptor). It was ever so simple really. It was metadata-driven. Take any relational database, point the V-Raptor setup tool at it, and within a minute you'd have a set of metadata describing the shape of the database, based on the database itself and some sensible defaults. You could then run the V-Raptor forms engine against this metadata, and you had an immediate set of CRUD forms, together with all the one-to-many and many-to-many relationships represented in a slick user-interface which leveraged server-side rendering (ASP) with browser technologies like HTML, CSS and JavaScript. This was a low-code/no-code solution before SharePoint, InfoPath, any other data-driven Intranet CMS software and before low-code/no-code was even a thing.

Of course you didn't have to live with the defaults. A power user could move things around, set validation rules, change the flow between forms, and give the entire system the same kind of richness which would have required huge development effort prior to the creation of V-Raptor. If you needed to change the underlying database, no problem  -  V-Raptor would adapt, and even make the changes for you. Compared to our previous ways of doing things, V-Raptor cut 90% off development timescales and budgets. Even custom form components cost far less time to develop, because they were now sitting within a framework is designed in a way which facilitated their creation. And best of all, once created they were available for all V-Raptor users and systems. And that's the sheer power of abstraction as an architectural technique.

We were a small company. We sold the technology and the company to our largest client, did our earn out, and established JFDI Consulting as our next business venture. We went on to build upon the concepts and create other implementations of the data-driven forms generator using new, faster, more secure technologies, deployed into codebases within other companies. It was a robust and sound idea borne of a desire to abstract out and generalise the commonality between systems. There are still systems out there now using our code, borne of the V-Raptor technology.

Processing the Processes

We had always wanted to create a workflow engine for V-Raptor. The idea was to generate events from the changing data, and use these events to fire rules which took actions if conditions were met. But despite a few forays into how we might achieve this, we never made significant progress before our company disappeared into oblivion within the self-destruction of its new owner.

Eventually technology changed, and new technologies emerged. One of these new technologies was The Internet of Things, or IoT. Like all emergent technologies, IoT caught our eye and we decided to take a look. What we found was lots of proprietary products with their own protocols. We found corporate politics and rivalries getting in the way of communication between product lines which were kept incompatible, even though they might be naturally mutually complementary. We found lots of data being generated by sensors but no over-arching technology for orchestrating the ingestion of that data and taking actions based on rules.

Yes, the same problem we'd come across 20 years earlier. So we got to work creating a solution. We started to abstract out all the commonality between rival product lines and make them work together. We needed a central data-ingestion engine to absorb the data, augment raw data with computed stats and contextual information, and generate events. We needed a rules engine to process rules which were simple enough for a power-user to create but with enough power to ensure their usefulness and flexibility. So we wrote each piece, and bit by bit Bobb was born.

Read more about Bobb here.

We always use our own technologies on ourselves first. Eating one's own dogfood, some call it  -  or dogfooding for short. My wife became one of the principal testers of Bobb. She was already familiar with some home automation tech like Philips Hue bulbs, Logitech Harmony remote, RF-controlled mains sockets and a Nest thermostat. She appreciated some of the conveniences; but she also noticed their shortcomings. Like how they mostly didn't work together, or indeed in many cases do very much. Our staff at the office also became testers.

So what exactly can an RF-controlled mains socket do, apart from turn a thing on or off? Well, connected to Bobb it can do a lot more. If it's always controlled by Bobb, Bobb knows when it's on or off. It therefore knows how long it's been on, and can implement timeouts. If Bobb knows something about what's plugged into the socket, it also knows how much energy it's used. Instead of just being controlled by the remote control it came with, Bobb can establish multiple user interfaces to control the socket, including voice AIs like Google Assistant, Siri and Alexa, or events like someone coming home. Maybe you'd like to control it using one of those wall switches you bought for your Philips Hue bulbs. With Bobb, you can.

We carried on merrily developing new ways for it to help us make our working and living environments better. New sensors generating new data feeding new statistics triggering new rules and controlling new devices in more and more ways.

But the thing about abstraction is that it makes one solution look dramatically more similar to another than it did before. Work out what's common, turn that into an engine, and each system just ends up using the engine in slightly different ways.

Hang on, we thought dramatically one day in a fit of inspiration, this isn't just an IoT orchestrator  -  it's the V-Raptor workflow engine. It doesn't matter what data comes into Bobb, it's just data. Financial data, sales data, manufacturing data, market data, sensor data  -  it's all ultimately the same. Bobb's data engine and rules engines are powerful and flexible enough to run a Smart Factory. They can also form the automation part of any business system where things need to happen based upon events generated from the shape of incoming data. So Bobb became a Smart Building orchestrator. Or a Smart Farming orchestrator. Or a Smart Metering platform, Business Process orchestrator or even a way to help generate data sets to support machine learning technologies.

We'd originally envisaged Bobb as a cloud-based system, architected for scalability so we could throw massive amounts of data at it and operate it as a multi-tenancy subscription service. But of course we then realised that we'd made it so flexible that we could run it on a small local device like an industrial-strength version of a Raspberry Pi (Bobb-Local), or spin up a special instance of it in the cloud to serve just one substantial customer (Private Bobb).

Thus, through abstraction we created a backbone technology which will become ubiquitous in every process we automate going forward. Whilst we though we were creating one thing, it turned out that by generalising a heap of the process automation we were actually creating something far more flexible and useful.

Coming Around Again

More recently we did the same again (but different) with Aardvark Sales for Teams, a collaborative xRM tool. Microsoft Teams is the collaborative core of Microsoft 365, representing a new way of doing work in a post-Intranet workplace. It's a place where business processes meet users face-to-face, and clearly that's always been an interesting battleground for us.

We'd seen a close business friend go through several partnerships with CRM software, ultimately finding them unsatisfactory in some way. Generally it was a lack of flexibility. People did what people always do: they see a useful tool and try to bend that tool to many different use-cases, often straining their capabilities way past comfortable. Custom development on top of a standard package is one way to give you a better fit to the unique shape of your organisation. But unless extensibility was built in from the beginning then the customisations will be incongruous or at worst unmaintainable. Standard packages generally had fixed design goals from their conception. And a fixed set of design goals with insufficient abstraction in the architecture of the solution will always end up hitting dead ends when you try to flex it too far.

We'd already had some experience creating this sort of software  -  from an early time we'd created many contact-management programs within other systems. Often we'd extended these with meeting notes and other data, some sort of categorisation for recording state within a process, and so on. It's the creation of this sort of core to many systems that led us to the V-Raptor turning point all those years ago. So when our friend came and asked us if perhaps we could create a better CRM system which drew its collaborative features from the host environment of Microsoft Teams, we went back to our old friend abstraction as a way of thinking about the problem afresh.

Treating the customer relationship lifecycle as a process made sense to us. But how many stages should form the process? And what data should we store for a customer? Moreover, where should we store the data?

Of course we couldn't come up with a one-size-fits-all data format for a customer. Or indeed the customer relationship lifecycle/process. And when the best answer to each question is "it depends", you probably need to think in a more abstract way. Of course we've been here before - V-Raptor was built as a data-driven forms-generator, keeping the underlying data structure up to the client, with the flexibility to conform the front-end to whatever data we're dealing with. So Aardvark came to incorporate the latest implementation of an old friend, developed using the best development tools of now.

As for where to store the data… Well, we set ourselves a couple of fixed rules to help us make that decision:

• The software had to work inside Microsoft Teams
• The data had to belong to the user even if they stopped using our software

If you use Microsoft Teams, you get some data storage for free. You can store your files in what are actually SharePoint document libraries, and those are just SharePoint lists with file attachments. SharePoint lists have all sorts of built in features which make our development workload smaller, including the ability to create views with multilevel sorting. So that's what we chose as our data store.

With the Microsoft Teams/SharePoint environment we get some other nice things for free too. Microsoft Flow allows users to trigger and automate extra processes based upon Aardvark Sales for Teams data. And we can even create PowerApps to feed key data into parts of the process using mobile technology, maybe on-site during sales meetings.

The task was to create a CRM/xRM tool for Microsoft Teams, with process-awareness. And we did, incorporating Microsoft Teams' collaborative feature set with a multi-phase Kanban-style categorisation framework. So a sales team can work together to progress leads through to closed sales; or a customer services department can manage customer support cases. But as it turns out, abstraction can inadvertently create so much more. When you allow Microsoft Teams to abstract out the facilitation of collaboration, and then represent stages in a process with buckets, you end up with something incredibly flexible  -  and applicable to many more use-cases than you originally envisaged. So watch this space for the forthcoming Aardvark HR Training Tracker, Aardvark Clinical Tests Manager and Aardvark Sort Out Your Life Step By Step.

I hope you like abstraction too

Abstraction requires the right thinking. You have to know the details intimately at a technical level, then be able to zoom out and see the big picture (how technologies fit together), then the even bigger picture including how business processes interact, and the way those interactions affect the business. And then zoom back down to the technical details, keeping the big pictures in mind. Ultimately you need to be aware that people will interact with business processes at many key stages, and design great user experiences to ensure that they'll want to do their job and keep the wheels moving.

Joel and I are fortunate; we seem to naturally gravitate toward this manner of thinking, and a multi-decade career has allowed us to get lots of practice and become pretty good at it. The next decade presents us with mature versions of so many new technologies for automation, and it's very exciting. But wherever we go with all this new tech, we'll always create flexible tools rather than single-use tools by keeping abstraction at the core of our architectural thinking.

First published on JFDI's blog.

Jon Silver is a former contributing editor of PC Pro Magazine and has been writing for publications internationally since 1989.