This post is either for developers who want an “at-scale” overview of modern architecture or for technical decision-makers looking at a great way to build a digital pipeline in this new composable world.

In modern web architecture, we are faced with the daunting task of composing headless sources together into a cohesive experience that feels like one system for all stakeholders. Many consider the roadblocks to be technical, but they are also organisational. This is mainly because there is no more centre of the universe - the origin server - that controls everything. Headless sources are API-first and need to be integrated to create a website or app. Without an origin server, content editors, marketers and developers alike need to connect to different systems to get things done. Mildly put, this is a struggle and, in the words of actual people I’ve worked with: a dumpster fire.

In this article

You will learn about the two things you need to create high quality, easy to manage, secure and performant front-ends that don't make a bespoke architecture or a monolith of modern tech.

1. Use the Jamstack with your favourite framework and host on your favourite CDN.
2. Behind it all, you have an orchestration platform that is vendor agnostic, has a killer SDK and gives all team members the ability to compose content without bothering developers.

Combining these two things will make your digital pipeline run smoothly and future proof the investments made. You can add legacy platforms as data sources if you have the right orchestration platform. You can slowly but surely transition away from them without doing a big-bang change offering big brands a safe path into the future.

The Jamstack: why it sits front and centre in modern architectures

Sites built with the Jamstack approach are a combination of static files generated by the CI/CD pipeline of your choice. Most dynamic stuff happens in the generation step, where the codebase reaches out to APIs and services to render all pages statically.

The performance gains alone justify this approach. There is no runtime page generation, and the app doesn’t have to connect to an origin server to figure out what content to serve. If you want to scale up for Black Friday, just put the static files in more places on the CDN edge, and all is well. Next to the performance gains, you also have a much more secure system. If your architecture has an origin server with all the knowledge of the system, that is a weak point for security.

You can add dynamic data as well. CDN providers have built the ability to run code on the edge, implementing serverless functions and edge workers.

A serverless function is a service that lets you run code without provisioning or managing servers. Serverless functions tend to be simple and do not have a state. They require an input, and they will give an output. After use, they will not stay up unless you request them to do that. The benefit is that the cost is low and that these functions are not running if not needed - consuming less energy. You can use them when compiling your Jamstack site, but they also work well at runtime.

Edge workers (as Cloudflare calls them) are pieces of code that live on the CDN edge close to the user and execute when a user visits a URL. This is ideal for reading cookie values and changing the stream of HTML that the CDN renders for that page. With this approach, you can dynamically manipulate what the user sees in their context while still serving a static page initially created by your Jamstack site. This approach is excellent for rendering personalised content based on user actions without the need for JavaScript hydration or an origin server.

The orchestration: how to cable manage your architecture

How you connect things up before the site is rendered is vital for the platform you are building. There are many technical and organisational problems to address when you scale a composed architecture to big organisations.

• Choices, choices, choices: CMS, Commerce, CDN, Marketing clouds, DAM, Personalization, A/B testing, CRM, etc.
• Integration is complex, and orchestration is hard to build.
• Often there is a lack of intuitive tools for non-devs to build and manage experiences.
• Early implementation decisions can limit the ability to change and evolve.
• Wrong choices can create developer bottlenecks and hinder time to market and agility.

The bigger the organisation, the more critical this statement becomes: To achieve true value, you need to orchestrate technology, people, and processes.

The digital pipeline can be stalled at many places. You’ll need a platform to help you orchestrate the whole architecture into a usable system for all stakeholders. A cohesive system has the ability for multiple systems to work together as one - for each system to have an awareness of what the other systems are doing. This same story goes for the practitioners in the system. From developers to content editors and from marketers to data analysts. All these people play a vital role in your online success.

The platform we need

The platform we need has to be “opinionless” and technology agnostic. Its primary goal is to cater to all stakeholders and prevent re-platforming.

Services integrators or agencies often use the “re-platform” model. Out with the old monolith and in with the new monolith. This way, they can sell their business transformation story and do a big bang release. The drawback here is that re-platforming takes a long time and is costly. It is also the complete opposite of an agile approach to a project. Agencies tend to be "platform partners" and sell in a platform first solution rather than a specific value solution to a brand. This is logical as they can hire specialists that retain their value project after project. However, modern architectures demand making choices for the business's needs, not what the agency or software vendors want to sell in. Square pegs and round holes could work for a while until it’s too hard to manage, and then you re-platform. Again…

The orchestration platform should support end-to-end delivery of digital products:

• Development: developers build components that incorporate content from multiple sources without building custom integrations.
• Authoring: practitioners build personalised digital experiences using no-code tools, including instant preview without involving developers.
• Deployment: developer and practitioner activities automatically trigger deployments to your CDN of choice to ensure digital experiences are always current.
• Delivery: Personalization and experimentation is delivered from the edge for the fastest possible performance.

The key is the “power of choice” for the tech stack, hosting / CDN, data fetching and what integrations are used. The point is that you as a technical stakeholder can choose when to add, remove or scale something without being held back by the system. There is no vendor lock-in, roadmap constraints, and re-platforming (swap an integration, migrate the data, and change some data mapping code).

Data fetching approaches

Flexibility in how data is fetched from different sources is vital for the longevity of an orchestration platform. The platform SDK should not apply any opinion on how data is fetched. It offers an SDK that helps developers fetch it the way they want.

It’s essential that page compositions only point to data sources and that the platform itself does not store anything except for the identification of the component it references. The benefits are:

1. the platform doesn’t store potentially sensitive data;
2. the platform doesn’t duplicate content;
3. the platform doesn’t have to know when data changes in the external source;
4. The platform does not have to serve the data and ingest the SLA provided by the source data it delivers.

It’s up to the developer to use the platform SDK to retrieve the compositional data and use platform provided helper tools to query the different API endpoints. The beauty of this approach is that the SDK and how the data is fetched can live anywhere. From the local codebase to an external middleware layer or a serverless function. There are some benefits to positioning the data fetching outside of your codebase:

1. You can hire separate developers who only focus on the API and the data mapping to the desired format. This separates concerns between disciplines, and back-end developers now have a place where they can feel at home.
2. The front-end application doesn’t need to know what external sources are queried. It queries one endpoint that returns the desired format for the front-end components to render.

A standalone front-end without knowledge of what external APIs feed it is highly flexible, future-proof, and hard to hack. Imagine a design system full of components with excellent documentation of what properties they need and a simple SDK to query data and map it directly to these properties. If you have to switch CMS or commerce engine, it’s a matter of remapping the data to the component properties and you are done.

Concluding

To successfully manage your digital pipeline, we need orchestration that adds structure to the stack, making it easier to align API first sources toward the same business goal and empowering all of the various teams contributing to the end product - developers, content authors and marketers. To keep the digital pipeline productive as new technologies emerge, you need orchestration that gives freedom of choice in every aspect of the architecture both today and tomorrow.

Imagine if marketers and developers could be friends again and work together to create the best experience for their website visitors?