Elixir: Building a Small JSON Endpoint With Plug, Cowboy and Poison

There’s been many times when I’ve just wanted to add a simple JSON endpoint to an app to, expose a service, or process webhook events, without the overhead of a full framework. Let’s see how easy it is to build a production ready endpoint with Plug, using Erlang’s Cowboy HTTP server.

Plug Is:

A specification for composable modules between web applications

Connection adapters for different web servers in the Erlang VM

If you’re coming from Ruby/Rails, think Rack, from Node, think Express, et al. Of course the concepts of these libraries are similar on the surface, they are unique in their own rights.

Cowboy Is:

A small, fast and modern HTTP server for Erlang/OTP

Additionally, it’s a fault tolerant “server for the modern web” supporting HTTP/2, providing a suite of handlers for Websockets and interfaces for long-lived connections. Without going into too much more detail, it’s safe to say, it’s an acceptable choice for production. Consult the docs for more info.

Poison Is:

A JSON library for Elixir focusing on wicked-fast speed without sacrificing simplicity, completeness, or correctness.

In other words, it’s a super fast, reliable JSON parsing library.

Building the Endpoint

With the short definitions out of the way, let’s build an endpoint to process incoming webhook events. Now, we want this to be “production ready”, what does that mean for our use case?

Fault tolerant: Always available. Can never go down (at least not easily :)
Easily configurable: Can be deployed to any environment
Well tested: Give us confidence in what we’re shipping

We do have a very simple use case for this, it’s a good idea to understand your own requirements before selecting tools and investing time into a solution.

1. Create a new, supervised, Elixir app:

$ mix new webhook_processor --sup
$ cd webhook_processor

--sup will create an app suitable for use as an OTP application. OTP and supervision will give us our #1 requirement from above. Our server will be supervised and restarted automatically in the event of a crash, while the server may crash the Erlang VM should not (at least not easily :).

2. Add Plug, Cowboy, and Poison as dependencies

# ./mix.exs
defmodule WebhookProcessor.MixProject do
  use Mix.Project

  def project do
    [
      app: :webhook_processor,
      version: "0.1.0",
      elixir: "~> 1.7",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      # Add :plug_cowboy to extra_applications
      extra_applications: [:logger, :plug_cowboy],
      mod: {WebhookProcessor.Application, []}
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:plug_cowboy, "~> 2.0"}, # This will pull in Plug AND Cowboy
      {:poison, "~> 3.1"} # Latest version as of this writing
    ]
  end
end

Couple notes here, we added plug_cowboy (in deps) as a single dependency for Plug AND Cowboy. We need to add :plug_cowboy to the extra_applications list (in application) as well.

3. Mix deps.get

$ mix deps.get

4. Implement application.ex

# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
  @moduledoc "OTP Application specification for WebhookProcessor"

  use Application

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: WebhookProcessor.Endpoint,
        options: [port: 4001]
      )
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

5. Implement WebhookProcessor.Endpoint

# ./lib/webhook_processor/endpoint.ex
defmodule WebhookProcessor.Endpoint do
  @moduledoc """
  A Plug responsible for logging request info, parsing request body's as JSON,
  matching routes, and dispatching responses.
  """

  use Plug.Router

  # This module is a Plug, that also implements it's own plug pipeline, below:

  # Using Plug.Logger for logging request information
  plug(Plug.Logger)
  # responsible for matching routes
  plug(:match)
  # Using Poison for JSON decoding
  # Note, order of plugs is important, by placing this _after_ the 'match' plug,
  # we will only parse the request AFTER there is a route match.
  plug(Plug.Parsers, parsers: [:json], json_decoder: Poison)
  # responsible for dispatching responses
  plug(:dispatch)

  # A simple route to test that the server is up
  # Note, all routes must return a connection as per the Plug spec.
  get "/ping" do
    send_resp(conn, 200, "pong!")
  end

  # Handle incoming events, if the payload is the right shape, process the
  # events, otherwise return an error.
  post "/events" do
    {status, body} =
      case conn.body_params do
        %{"events" => events} -> {200, process_events(events)}
        _ -> {422, missing_events()}
      end

    send_resp(conn, status, body)
  end

  defp process_events(events) when is_list(events) do
    # Do some processing on a list of events
    Poison.encode!(%{response: "Received Events!"})
  end

  defp process_events(_) do
    # If we can't process anything, let them know :)
    Poison.encode!(%{response: "Please Send Some Events!"})
  end

  defp missing_events do
    Poison.encode!(%{error: "Expected Payload: { 'events': [...] }"})
  end

  # A catchall route, 'match' will match no matter the request method,
  # so a response is always returned, even if there is no route to match.
  match _ do
    send_resp(conn, 404, "oops... Nothing here :(")
  end
end

This looks like a lot, but most of this file is just some helpful comments. The gist is, we are using the macros, get and post, from Plug.Router to generate our routes. This module is a plug itself, and it defines its own plug pipeline. Note, match and dispatch are required in order for us to handle requests and dispatch responses. Pipeline is a key concept here, as the order of plugs determines the order of operations. Notice that match is before we define our parser, this means we will not parse anything unless there is a route match. IF the order was reversed, we’d be trying to parse requests regardless of routes matching. Refer to the docs on Plug.Router for more info.

6. Make the Endpoint Configurable

# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
  @moduledoc "OTP Application specification for WebhookProcessor"

  use Application

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: WebhookProcessor.Endpoint,
        # Set the port per environment, see ./config/MIX_ENV.exs
        options: [port: Application.get_env(:webhook_processor, :port)]
      )
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

We’ve swapped out the hard coded port value of the Cowboy options for an environment variable, this will allow us to run the webhook processor in any environment we need to. Finally, create a config file for each MIX_ENV you need:

#./config/config.exs

# This file is responsible for configuring your application
# and its dependencies with the aid of the Mix.Config module.
use Mix.Config

import_config "#{Mix.env()}.exs"

-------------------

# ./config/dev.exs

use Mix.Config

config :webhook_processor, port: 4001

-------------------

# ./config/test.exs

use Mix.Config

config :webhook_processor, port: 4002

-------------------

# ./config/prod.exs

use Mix.Config

config :webhook_processor, port: 80

7. Test

# ./test/webhook_processor/endpoint_test.exs
defmodule WebhookProcessor.EndpointTest do
  use ExUnit.Case, async: true
  use Plug.Test

  @opts WebhookProcessor.Endpoint.init([])

  test "it returns pong" do
    # Create a test connection
    conn = conn(:get, "/ping")

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response and status
    assert conn.state == :sent
    assert conn.status == 200
    assert conn.resp_body == "pong!"
  end

  test "it returns 200 with a valid payload" do
    # Create a test connection
    conn = conn(:post, "/events", %{events: [%{}]})

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 200
  end

  test "it returns 422 with an invalid payload" do
    # Create a test connection
    conn = conn(:post, "/events", %{})

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 422
  end

  test "it returns 404 when no route matches" do
    # Create a test connection
    conn = conn(:get, "/fail")

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 404
  end
end

These tests are pretty simple, but they confirm our server is working as expected. One could argue that the only thing we should care about from these tests are the response codes, not the side effects that happen when events are processed. Always test UP TO the boundaries of your module, never beyond, unless you are writing an integration style test.

Conclusion

With very little effort we’ve built a small but mighty endpoint. Thanks to Cowboy, you should be able to serve more connections from one server than you’ll likely ever need, so let’s add low cost to the list of benefits as well.

What about deployment? Let's walk through building releases and deploying to AWS:

As always, the code is available on GitHub: https://github.com/jonlunsford/webhook_processor

Blog