Improve Rspec Pipeline Performance

At Mekari, we highly encourage that developers should write tests on every projects. Testing prevents bugs in production, forces better code, and encourages developer confidence when making changes to existing code bases. We use rspec, selenium with capybara as our standard unit tests and integration tests tools.

Rspec best practice

If you're new to ruby on rails and never used rspec before, here are some links which will give little guidance to start.

Working with Large Code Bases

In a large code base, testing becomes even more important. While some argue that testing slows development down, we argue that it increases development velocity over time, especially when refactoring existing code and eliminating technical debt.

Test Driven Development(TDD) encourages running your tests frequently and often. What you might find in a large code base when running the full test suite are it may takes 25+ minutes to run 2500+ tests. That will make you unproductive when you push changes and wait test suite to finish.

In this post, I will share how we accelerate our pipeline test. For my project, we use dedicated CI (Continuous Integration) server bitbucket for running the full test suite. Every minutes, developers make changes on a branch, commit, and push changes to bitbucket. Then, create PR (Pull Request) regarding ticket and bitbucket will pull down these changes and run the entire test suite.

Increase Hardware capability

When we want to achieve performance without doing much, our first option come to improve hardware specifications. You can easily see double the non parallel performance on your test suite by upgrading your CI server from an old or low spec machine into faster one.

Parallel Tests

Running specs across multiple CPU cores can further decrease the test time for your suite. We use parallel_tests gem for this. It uses a database per thread and can run on CI servers like bitbucket. Running test suite in parallel also make hardware upgrades even more effective, scaling out to machines with more cores. Single core performance is no longer your bottleneck.

When parallelizing your suite, beware of shared state between tests that lives outside of the database such as Elasticsearch. These interdependencies need to be addressed too in parallel.

This solution solve your issue about slow running your test suite, but they come at a price.

Profiling The Specs

When you have implemented the above solutions, but you still feel slow with your test suite. Then, what you should try to consider is your database strategy setup and test suite.

Database Cleaner configuration

If you write ruby or rails code that interacts with a database and write unit test, chances are you have heard of or used the database_cleaner gem. It's a terrific gem that abstracts away the various ORM APIs for getting the DB into a "blank slate" state. Some setup needed to use it in optimal mode. Here are some article that give you idea to setup your own:

Factory cascade

Let's say you have User model and each user has one Contact and one Location record. Your factory should be looking similar like this:

factory :user do
  contact
  location
end

Now, try to guess how many records are created in the database once you call create(:user)?

Each time you call create(:user), then 2 additional records are created. Yes, even with build. Now imagine that you are using this base factory across all tests. There is 2 ways to eliminate this issue.

Explicit associations The first thing that comes to mind is to remove all (or almost all) associations from our factories:

factory :user do
  # do not declare associations
  # contact
  # location
end

With this approach, you have to explicitly specify all required associations when using a factory

create(:user, contact: contact, location: location)

# But!!
create(:user) # => raises ActiveRecord::InvalidRecord

Traits Your base factory should always have only these attributes that are needed to pass validation. If you sometimes need associations then consider using traits

factory :user do
  name { "Zuned #{Faker::Name.unique.name}" }
  username { "#{ name.parameterize.underscore}_#{ SecureRandom.hex(3) }" }
  email { "#{ username + Random.rand(1000).to_s }@example.com"}

  trait :with_location do
    location
  end
end

and you can call your factory using

create :user, :with_location

let_it_be and Anyfixture

Our specs always have relations or required company and 2 or more employees (with association to user, job title, location, and job level). Imagine if we call create :company and so on for every specs. 1 spec need 1 company + 2 employee + 2 locations + 2 job_title + 2 job_level, it means 9 records only for initial data. Can we generate those 9 records only once for every specs?

Yes, here comes anyfixture. Built in fixture are different with anyfixture. Both represent different approach. Fixture declare a static state of the data that is loaded into a test database right away and usually persists between test runs. But with anyfixture, we able to hard reload record to get fresh copy of record via refind method.
Here are our implementation with (Anyfixture)[https://test-prof.evilmartians.io/#/any_fixture].

require 'test_prof/any_fixture/dsl'
require 'test_prof/ext/active_record_refind'

using TestProf::AnyFixture::DSL
using TestProf::Ext::ActiveRecordRefind

# Fixtures are kept to be outside db transaction such that it can be leveraged
# for specs. https://github.com/palkan/test-prof/blob/master/docs/any_fixture.md
RSpec.shared_context 'company with staffs' do
  before(:all) do
    # register to fixture
    fixture(:company) { FactoryBot.create(:company) }
    fixture(:location) { FactoryBot.create(:location, company: fixture(:company)) }
    fixture(:job_title_ceo) {
      FactoryBot.create(:job_title, :ceo, company: fixture(:company))
    }

    fixture(:job_title_hr) {
      FactoryBot.create(
        :job_title,
        title: 'HR Manager', parent: fixture(:job_title_ceo), company: fixture(:company)
      )
    }

    fixture(:job_title_staff) {
      FactoryBot.create(
        :job_title,
        title: 'Staff', parent: fixture(:job_title_ceo), company: fixture(:company)
      )
    }

    fixture(:org_general) {
      FactoryBot.create(
        :organisation_node,
        name: 'General', company_id: fixture(:company).id
      )
    }

    fixture(:org_hr) {
      FactoryBot.create(
        :organisation_node,
        name: 'HR', company_id: fixture(:company).id
      )
    }

    fixture(:job_level_ceo) {
      FactoryBot.create(:job_level, :CEO, company: fixture(:company))
    }

    fixture(:job_level_hr) {
      FactoryBot.create(:job_level, name: 'HR', company: fixture(:company))
    }

    fixture(:job_level_staff) {
      FactoryBot.create(:job_level, name: 'Staff', company: fixture(:company))
    }

    fixture(:owner) {
      FactoryBot.create(
        :employee, :owner,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_ceo),
        organisation_node: fixture(:org_general),
        job_title: fixture(:job_title_ceo)
      )
    }

    fixture(:hr_admin) {
      FactoryBot.create(
        :employee, :hr_admin,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_hr),
        organisation_node: fixture(:org_hr),
        job_title: fixture(:job_title_hr)
      )
    }

    fixture(:staff) {
      FactoryBot.create(
        :employee, :staff,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_staff),
        organisation_node: fixture(:org_general),
        job_title: fixture(:job_title_staff)
      )
    }
  end

  # Ensure that every example has a fresh copy of a record
  %i[
    company location job_title_ceo job_title_hr org_general org_hr
    job_level_ceo job_level_hr owner hr_admin staff
  ].each do |object|
    let(object) { fixture(object).refind }
  end
end

Imagine this example too.

describe User do
  let!(:user) { FactoryGirl.create :user }

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end
end

How many User records are created in the database? One record for each example so the answer is 3 records. To avoid it you can use (let_it_be)[https://test-prof.evilmartians.io/#/let_it_be]. This helper uses Rails transactional tests feature which means that the record is created only once at the beginning of the test and removed when the test is finished.

describe User do
  let_it_be(:user) { FactoryGirl.create :user }

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end
end

Additional suggestion and resource

There are some condition where we doesn't need data to persists first into db like checking custom validation on model and unit test on view or mailer specs. Then if this happer, we can utilize build and build_stubbed create strategy.

let(:company) { FactoryBot.build_stubbed(:company) }

some usefull links:

(Test Profiling)[https://test-prof.evilmartians.io]
(Factory therapy for your ruby tests)[https://evilmartians.com/chronicles/testprof-2-factory-therapy-for-your-ruby-tests-rspec-minitest#bonus-anyfixture]
(build and build_stubbed)[https://thoughtbot.com/blog/use-factory-girls-build-stubbed-for-a-faster-test]

Blog