Do you have a Ruby on Rails app that manages integrations with multiple 3rd party APIs or external data sources?

Do you struggle with keeping your application's domain logic separate from these external data sources and flows?

Do you feel like you have to customize main flows in your app for each data source or API?

Can I grow a few extra arms to raise my hands twice for all this? ✋✋✋

Here's what I'm going to cover in this post:

• How this became a problem
• The Service Object architecture we've committed to for untangling these dependencies (aka Skip to the Code)
• 2 months later - how it's been going
• The resources that were the most helpful to learn from and adapt for our use case (jump to resources)

The Problem: The Snowball Effect

I work on an eCommerce app that takes external data from our "sources" and makes that data available to our customers via a catalog. The value of our product is that it takes the data from multiple sources and combines it into one catalog.

To simplify the examples to come, let's think of the product as an online grocery store. Each data source is the maker of an item you can purchase in a grocery store. So you have farmers that supply the produce and cereal makers who supply the cereal, and the people who make Eggo waffles, etc.

So where we got into trouble with our app is that (perhaps unsurprisingly) each data source formats their data differently. They name things differently, they have different ways of matching up a SKU (an individual thing you can buy, like a snack-size pack of Oreos vs. the family-size pack of Oreos), and just think about The Thing They Are Selling differently from other suppliers.

We started out with what we call "integrations": a directory of namespaced code that houses the fetching of the external data and translating it into the common language of our Grocery Store offerings. It looks a little like this:

 app/
  |__ integrations/
    |__ farmers/
      |__ fetch_data.rb
      |__ format_data.rb
      ...
    |__ cereal_makers/
      |__ fetch_data.rb
      |__ format_data.rb
      ...
  |__ catalog/
    |__ update_skus.rb
    ...
  ...

At first, this was great! We took in the new data, structured it how we wanted, filled in missing data with default values, sanitized data - no problemo. But then we needed to add another integration. And - guess what?! - they didn't have an API. So now we had to make a CSV upload process that handles fetching this new integration's data. But we still wanted the same output: a catalog of consistent data across all suppliers.

And then we wanted to add yet another integration...and our minds sort of exploded, to be honest 🤯 . You had to be intensely familiar with each integration's incoming data structure and know THEIR domain-specific context like whether they use an API or CSVs to send us their data. And you had to support multiple intake processes that could all break in different ways. And we needed to funnel all this data to one, common process to populate our catalog.

The Solution: Rails Engines + Structured Service Objects

I'm not going to spend much time talking about Rails Engines (that deserves a post of its own!) but there's pretty good documentation out there about it if you want to learn more. The examples to follow don't require you to also use engines.

Isolate Code Related to External Logic

So the first change we committed to is that each integration with a different data source will have its import code live inside of a Rails Engine (we call them Components). A hand-wavey definition of an engine is ruby gem that is internal to your app. It has access to the main app's code, but has its own routing, test files, services, etc. For my team, this was an easy way to draw a line in the sand and say "if the behavior I'm working on belongs to the data source and not our core app, it goes into the engine".

Our app structure doesn't look much different now:

 app/
  |__ components/ # moved these integrations into a "components" directory
    |__ farmers/
      ...
    |__ cereal_makers/
      ...
  |__ catalog/
    |__ update_skus.rb
    ...
  ...

but our main "Core" app Gemfile now looks like this:

# Gemfile
...
gem 'farmers', path: 'components/farmers'
gem 'cereal_makers', path: 'components/cereal_makers'

Now that we've pulled our integration code out of our core app, let's get down to the nitty gritty with dependency management inside our components!

Service Objects & Dependencies

To establish this pattern, we needed to commit to some non-negotiable expectations:

• Service objects will always return a "response object" with a specific shape
• Service objects maintain their own dependencies
• Service objects should handle their own failure as well as surface the failures of any dependencies

Here's what our service should look like at the end of this process:

# app/components/famers/app/services/farmers/lettuces/create_service.rb
module Farmers
  class Lettuces
    class CreateService < BaseService
      self.build
        new(Farmers::Produce::InventoryManagerService.build)
      end

      def initialize(manage_inventory_service)
       @manage_inventory_service = manage_inventory_service
      end

      def call(lettuce_params)
        @lettuce_params = lettuce_params

        lettuce, created = create_lettuce
        inventory = @manage_inventory_service.call(lettuce)

        OpenStruct.new(
          created?: created,
          lettuce: lettuce,
          inventory_updated?: inventory.updated?,
          errors: [lettuce.errors, inventory.errors].join(', ')
        )
      end

      private

      def create_lettuce
        lettuce = Lettuce.new(@lettuce_params)
        if lettuce.save
          return [lettuce, true]
        end
        [lettuce, false]
      end
    end
  end
end

Expectation: Service objects will always return a "response object" with a specific shape
This is a personal decision for your team based on your own experiences and preferences. We have a completely separate frontend app and have struggled with error handling before. So for us, we wanted to always have our controllers return status: :ok but have json objects that include errors if they exist.

We decided upon a structure with these guidelines:

• The mutated/created record is returned (usually with a key of the model name, like lettuce: lettuce)
• There is a boolean status key that indicates if the service completed its job or not (could be success?: true or created?: false)
• There is an errors key that returns the error message(s) or nil

We used an OpenStruct object, which is mutable (meaning a service could set OpenStruct.new(success?: true) and then elsewhere it could be overwritten OpenStruct.new(success?: false). This hasn't been an issue for our team, but is good to keep in mind.

Service objects maintain their own dependencies
This falls under the larger consideration of "how the heck do I manage all this state?!".

# app/components/famers/app/services/farmers/lettuces/create_service.rb
module Farmers
  class LettucesController < ApplicationController
    def create
      results = Lettuces::CreateService
        .build
        .call(lettuce_params)
      ...
     end
    ...
  end
end

# app/components/famers/app/services/farmers/lettuces/create_service.rb
module Farmers
  class Lettuces
    class CreateService < BaseService
      # bad
      def call(lettuce_params)
        @lettuce_params = lettuce_params

        lettuce, created = create_lettuce
        inventory = Farmers::Produce::InventoryManagerService
                      .build
                      .call(lettuce)
        ...
      end

      ...
    end
  end
end

# app/components/famers/app/services/farmers/lettuces/create_service.rb
module Farmers
  class Lettuces
    class CreateService < BaseService
      # good
      self.build
        new(Farmers::Produce::InventoryManagerService.build)
      end

      def initialize(manage_inventory_service)
       @manage_inventory_service = manage_inventory_service
      end

      def call(lettuce_params)
        @lettuce_params = lettuce_params
        ...
      end

      ...
    end
  end
end

Two Months Later: How It's Been Going

I can honestly say we've been loving this change! We've started converting over old service objects and creating all new ones in this style. It's made it so much easier to know:

• what is going to be changed/done when I call this service
• how will I know if it succeeded or failed?

It's made us become much more thoughtful about single-purpose services that don't need to be so generalized and vast. They can be specific and accurate to one case and then put together to handle generalized cases.

Recommended Reading

This blog post by Adam Niedzielski is the main influencer of how we handle dependency injection in Service Objects.

• It had the best examples of multiple services that rely on each other
• There is a github repo - with tests! - that give a more in-depth view on this approach

This blog post by Dave Copeland at Stitch Fix has some really great narrative about handling this same problem in a really big app with lots of developers working on the project.

• Includes some insight to how they created a gem to enforce immutable (meaning it can't be changed after the fact) service object responses
• Is opinionated (in a good way) about good vs. bad practices with service objects

This blog post on Toptal by Amin Shah Gilani is actually one of the first articles I read when searching for "a better way" of handling service objects. It ended up being a different path than we committed to, but it still a good generalized read about Rails Service Objects.

• Includes opinions on where a service object should live in your app
• Great example code
• Thoughts on what a service object should return