Event-Driven APIs with Webhook and API Gateway

There are many ways and technology options to consider when implementing an event-driven API. For example, we explored how to build event-driven APIs using these 3 well-known patterns: CQRS, API Gateway and Serverless on the previous blog post. This post elaborates on building event-driven APIs by making use of Webhook and API Gateway, we understand the role of each in this solution. Firstly, let’s turn our attention to the initial problem statement without the webhook in place.

Need for a webhook

Consuming applications expect to be informed of any change of state on a specific record or records.

Examples:

Updating or adding customers in a CRM system triggers an event.
Currency exchange rates from a foreign exchange application informs users about currency change.
New post on a user's blog notifies subscribers.
New order creation on an online shopping application informs another service in the system.
An orchestrator service wants to be notified when Service A completes a task and when to handover the task to Service B in a data ingestion pipeline and so on.

The majority of APIs only support these types of requirements by having the consuming application constantly poll for changes. This means that the consuming application has to make frequent API calls to find out any changes of state in the desired resource. This is highly inefficient, and calls may result in empty payloads when there haven't been any updates. Also, what if the called HTTP API accepts our HTTP request but takes a long time to handle it, this could affect the user experience, especially when the behavior is reflected in the user interface (meaning a user has to refresh a page to get the latest changes).

Instead of having to constantly poll for changes, create a subscription endpoint against a specific resource so consuming applications can register their interest to be informed on any change of state (an event) by providing a call-back endpoint. At this point, it becomes the API's responsibility to send back any change of state by posting the updates to the registered endpoint.

What’s Webhook?

A webhook is a software architecture approach that allows applications and services to submit a web-based notification to other applications whenever a specific event occurs. The application provides a way for users to register or connect API calls to certain events under specific conditions, such as when a new user, account, or order is created or an order ships out of a warehouse.

Webhooks are generally used to notify clients of events, in real-time, as they occur. They normally take the form of HTTP POST endpoints that can be requested with a JSON body and it is fully managed by an event consumer. An event producer, such as an API server, can send event notifications to a webhook when something interesting happens.

Webhook and API Gateway in Event-Driven Architecture

Leveraging Webhook and API Gateway enables you to build an event-driven API that can be decoupled from your main application code. Enabling you to call external systems that have subscribed via webhooks in complete isolation from your application code.

As you can see in the preceding architectural diagram, there are two main flows.

Subscription process

In the first flow on the left, API Consumers can subscribe to the API by registering a Webhook URL as the callback. A consuming application subscribing for changes in a resource by making a POST call (with the call-back URL in the body) to a resource subscription API endpoint (for example, /{resource}/subscribe) exposed in an API gateway. Once the API gateway receives the call, it routes the request to the subscription service, which then adds the subscriber details to a database.

It is also possible to unsubscribe from the API. In this scenario, API Gateway’s tasks first identify unknown messages so be sure that the request is always authenticated and credentials are valid, it returns a 2xx response immediately as an acknowledgment or if the request cannot be authenticated or there is an error getting the payload into a staging system, an error is returned. Then it is also passing the request to the responsible service based on the path provided by the consumer.

Callback process

In the second flow on the right, we are delivering events to API consumers asynchronously through the call-back component and the API Gateway. An event listener service queries the database as subscribers are matched against particular processed events. The event listener service then creates call-back commands and publishes them in an Event Hub so a call-back service can execute all API calls (and retries if necessary) via the API gateway.

There, API Gateway plays not only a role of a reverse proxy but can convert internal calls from one format to another. For example, the call-back service is using another AMQP (Advanced Message Queuing Protocol) messaging protocol but the API should make a REST call to the consumer’s callback endpoint, in this gateway an API Gateway such as Apache APISIX can help. It can receive a REST request, then transform it to the desired format and forward it to a service, get the response, and return it to the client in REST format utilizing its different plug-ins.

Also, it comes with other concerns like securing it with certificates and preventing DDoS attacks. And it enables a monitoring feature for your webhook to be able to see what is going on with the webhook, you know like what is wrong with the configuration on the API provider side.

One of the most efficient ways to handle the webhook processing part of the above architecture by using API Gateway of your choice and an event-driven serverless function.

Summary

As we understood throughout the post, Webhook tries to decouple the concerns like a message acknowledgment and the processing messages in the API and no synchronous business logic is performed. However, the above architectural example we discussed can be a complicated pattern to implement given that it has many moving parts and the API are not aware of a consuming application endpoint is up and running but that can be improved. In addition to this, Webhooks force the event consumer to establish a publicly accessible HTTP endpoint to receive events that are not secure enough. We can secure it properly by enabling some authentication mechanism (Basic auth, JWT or other ways) with the API Gateway capabilities.