I have recently joined a new greenfield project that requires complex and business defined process flows. These flows have multiple calls with external services, business operations approvals, four eyes checks and interfaces with vendor solutions. To meet these requirements we determined an orchestration layer was required, allowing us to coordinate these flows and provide necessary recovery logic. However, to develop this service without any reasoning tools could lead to a ballooning system making comprehension of the underlying requirements and flow difficult. Due to the nature of our project we determined that high observability was key as we wanted to involve all stakeholders in the development process. Due to this we found BPMN diagrams and specifically BPMN engines a vital tool in the definition and development process.

BPMN (Business Processing Modelling Notation) diagrams are a diagrammatic method of modelling business processes. With standardised symbols to make process flows easier to understand for all relevant parties. It provides a method of defining complex process semantics, yet in a format that is still comprehensible and intuitive to business users. BPMN diagrams allow all stakeholders the ability the use a common language to discuss process flows, unlocking a collaborative approach to the definition and review process.

There are 4 main elements in a BPMN diagram, these elements together can define anything from abstract processes, to detailed process flows, to implementation ready processes.

There are plenty of examples of complex, real-world, technical BPMN solutions. This diagram defines the flow for a Chinese takeaway ordering system and demonstrates the use of the elements above. While this is a high level example we can start to see where a developer may be able to find use in this diagram when it comes to the flows implementation.

Using BPMN diagrams in the definition and development cycle is valuable due to its ability to bring together business analysts and technical developers using a visual medium. However, once complete these diagrams are still just diagrams on paper. From here developers have a reference that is used to implement the defined flows. This introduces a few issues and questions. It takes time to implement these processes and once complete it can be hard to prove the implemented solution is a 1-1 match to the business defined flows. Furthermore, the traceability of the solutions often rely on logs of processes, loosing the common and visual solution offered by BPMN diagrams. While BPMN diagrams are brilliant to get an initial process definition defined, together with all stakeholders, we need to have a way to keep this universality through to implementation and even production.

Step forward Camunda, Kogito, Azure Logic Apps and other BPMN engines. These tools can be integrated directly into or alongside a systems orchestration service. After interpreting the BPMN model these engines are also able to execute it. Meaning the engine creates an instance of the defined process, evaluates to flow conditions as it reaches each gateway and moves the instance through the diagram according to its definitions. The engines will provide all functionality in the OMG BPMN standards, including: assigning users tasks, waiting for approvals, send emails, query services, retry on failure, and much more.

Now our BPMN diagrams define and control the implementation at the code level. We can produce delegate blocks with developer defined logic where complex steps are required, but the overall flow that instances must follow is defined by the BPMN diagram. This has a multitude of benefits and often these engines come with extra treats to make this even more valuable. The first clear advantage is observability. Now this diagram is a 1-1 mapping with the actual logic our code will follow and we can be certain that every instance will have the necessary checks or enrichment required during the initial definition.

With the BPMN diagrams integrated into the actual flow of code, BPMN engines like Camunda (my teams preference) can provide auxiliary features. Camunda itself easily integrates with Java SpringBoot, meaning a complex system can be built around the engine to supplement any required delegate logic. The Camunda Engine also provides a feature called Camunda Cockpit which is a UI available while the engine is running where instances running through the process flows can be tracked and evaluated. This means you can follow the execution of the process and observe the values passed through the flow. In fact in Camunda and many other engines we can follow these instances through the flow on the diagrams in real time (more on this later).

Camunda also comes wrapped with many resiliency features that are required by a central orchestration system. With local and global replication built-in to provide seamless replacement of faulty nodes. One feature provided by Camunda is External Tasks which is a topic-based queue system for communication and coordinating worker nodes. This will store and manage the task ready for execution with priorities and locking policies, but no code is required! The engine interprets the requirements from the diagram and converts this to a complex and resilient message queue.

With such a powerful engine we can also start to collect metrics from our UAT and PROD instances. The features in Camunda Optimize (WARNING: requires the enterprise version) offers insights into the BPMN processes. For instance we can observe a heat-map of all flows, highlighting either common waits for execution time or the most executed stages. Camunda Optimize also provides branch analysis to help visualise and compare variations on process flows.

BPMN diagrams are designed to bring all parties together to define business logic using a universal, graphical language. I have found that incorporating a BPMN engine into the heart of my teams system has enabled for further inclusion in the development journey. Allowing for more accessible demos and has lead to better feedback and engagement. This collaboration is at the heart of a high-functioning, agile team and with Camunda and BPMN diagrams we have drastically reduce the time it takes to design and agree upon process changes.

As teams become more integrated and less siloed, I believe that as developers we will have to find better ways of communicating and collaborating with non-technical team members and even customers. I have found BPMN diagrams and engines a perfect tool for this purpose and hope to continue towards a more unified business journey.