Honeybadger Staff
Posted on October 5, 2022
This article was originally written by William Kennedy on the Honeybadger Developer Blog.
Rails is a framework that comes with nearly everything included, focusing on conventions over configurations. Minitest is one of these conventions. Minitest is small and fast, and it provides many assertions to make tests readable and clean.
However, many alternatives to Minitest are available. The most popular one is RSpec.
RSpec has the same goals as Minitest but focuses on readable specifications describing how the application is supposed to behave with a close match to English. Minitest purports that if you know Ruby well, it should be enough.
The RSpec project focuses on behavior-driven development (BDD) and specification writing. Tests not only verify your application code but also provide detailed expressions that explain how the application should behave.
Minitest also supports test-driven development (TDD), BDD, mocking, and benchmarking. However, they have subtle differences, so we’ll go through an example app in both RSpec and Minitest to demonstrate the differences. Before we begin, let’s cover what all testing frameworks have in common.
Let’s Take a Step Back. What Is Testing?
Both TDD and BDD encourage a test-first approach, which means that you start by writing the test. You run the test, which should fail. Then you write the code that makes the test pass.
By writing a lot of tests to cover minor aspects of our overall program, we should eventually arrive at a nearly perfect system. Of course, this is software, so that doesn’t happen in reality. Yet, we can make drastic changes to our software with reduced risk by having many tests. Automated testing can be the difference between success and failure in a small startup pivoting to find market fit. In a large company, innovation does not have to slow down.
What are tests? We can answer this question by breaking testing down into three broad categories in the software world.
Unit Tests
A unit test is a small, automated test coded by a software developer to verify whether a small piece of production code – a unit – works as expected.
Many essays have been written about unit-tests, but one acronym you should be aware of is FIRST, which was first used by Tim Ottinger and Jeff Langr:
- F - Fast - Tests must be fast to run. The faster they run, the more you’re likely to run them.
- I - Isolated - Each test should have a single reason to fail
- R - Repeatable - Tests should have the same result every time you run them.
- S - Self-verifying - Results should not be open to human interpretation. They should be binary (e.g., Red/Green).
- T - Timely - Write the tests before you write the code.
Integration Tests
The next broad category is integration tests. These tests verify that small pieces of code work.
Therefore, we can determine that A, B, C and D unit tests work independently, but how do you know that they work together?
Integration tests can sometimes be difficult to write and may be created as part of the bug-fixing process.
UI Testing
This branch of testing generally has the highest compute costs and runs a simulated user experience to test bugs, flows, and everything else. It’s most often used to replace or assist professional Quality Assurance testers.
Now that we’ve broken down software testing into broad categories, let’s go a step further and explore another important concept before comparing RSpec and Minitest.
Red, Green, and Refactor Lifecycle
In one of my first development jobs that practiced TDD, the senior developer used the three 'Gs'.
“Get it failing, get it working, and then get it Better.”
This made our workflow look like this:
Essentially, we write a test first then run it. The test will fail. Next, we write the code that makes the test pass. After the test passes, we can move on and improve the implementation. Slowly but surely, we arrive our ideal implementation.
This simple workflow seems counterintuitive because it is slower initially. However, in the long term, it is faster.
Comparing Minitest to RSpec
Chances are that if you are working on an existing codebase, the decision has already been made for you. If you are starting a project, you might be wondering if you should explore using RSpec or stick with Minitest. If you’re wondering if you should switch from one to the other, this is more nuanced and, depending on the size of your codebase, might not be worth it.
I hope you will see the actual code you will be writing, which will help your decision-making.
Here are the topics we’ll compare
Setup - Minitest Vs RSpec
To begin, we’ll create two similar apps and set up RSpec and Minitest.
Since Minitest comes installed with Rails, it’s simply using the Rails new command.
rails new rails_minitest
However, when setting up RSpec, we have to do more.
rails new rails_rspec
After the app is generated, we have to add the RSpec gems. This is where we encounter the first difficulty with RSpec.
Do we add the RSpec core gem or the rspec-rails gem?
It might seem obvious to select rspec-rails since we’re dealing with a Rails app, but this would be a mistake that I've seen more than once in my time as a consultant.
Setting up rspec-rails involves adding it to our Gemfile in two locations.
The project's readme covers all this.
# Gemfile
## For all the generators
group :development, :test do
gem 'rspec-rails', '~> 5.0.0'
end
After adding this to your gem file, you can do the following:
bundle install
After installing, we can then run the rspec:install command.
rails generate rspec:install
This will create a spec folder, a .rspec file, and two additional files, spec/spec_helper.rb and spec/rails_helper.rb.
What is the difference between these two files?
The spec/rails_helper loads the Rails app, and spec/spec_helper is a lightweight configuration file for RSpec.
As a quick aside, you can speed up some of your RSpec tests if you avoid loading Rails for particular files. For example, if you are using plain-old Ruby objects(POROs) in your app that don’t need Rails, you can avoid adding Rails to that spec when running the test. This speeds up the feedback loop when writing the code and helps with the F(ast) in our FIRST principle, explained by Tim Ottinger and Jeff Langr.
Here is an example:
# app/services/hello_world.rb
class HelloWorld
def say
'Hello World'
end
end
# spec/services/hello_world_spec.rb
# Usually, we require 'rails_helper' here, but there’s no need if we are not using Rails.
require './app/services/hello_world'
RSpec.describe HelloWorld do
describe '#hello' do
it 'returns hello world' do
expect(HelloWorld.new.say).to eq('Hello World')
end
end
end
Unit Testing: RSpec vs. Minitest
Now that we have set up our testing suite, we can compare how we write tests in each framework.
Let’s look at an example model test in RSpec. The convention is to write the test in plain English then in code.
require 'rails_helper'
RSpec.describe Article, type: :model do
context 'validations' do
article = Article.new
article.valid?
it 'must have a title' do
expect(article.errors.messages[:title]).to include("can't be blank")
end
it 'must have a body' do
expect(article.errors.messages[:body]).to include("can't be blank")
end
end
end
Run the test:
rspec --format documentation spec/models/article_spec.rb
When the test fails, we get the following output:
Article
validations
must have a title (FAILED - 1)
must have a body (FAILED - 2)
Failures:
1) Article validations must have a title
Failure/Error: expect(article.errors.messages[:title]).to include("can't be blank")
expected ["is too short (minimum is 5 characters)"] to include "can't be blank"
# ./spec/models/article_spec.rb:9:in `block (3 levels) in <top (required)>'
2) Article validations must have a body
Failure/Error: expect(article.errors.messages[:body]).to include("can't be blank")
expected [] to include "can't be blank"
# ./spec/models/article_spec.rb:13:in `block (3 levels) in <top (required)>'
Finished in 0.02552 seconds (files took 1.3 seconds to load)
2 examples, 2 failures
Failed examples:
rspec ./spec/models/article_spec.rb:8 # Article validations must have a title
rspec ./spec/models/article_spec.rb:12 # Article validations must have a body
To get this output, you must run the RSpec command with the —format documentation option. The default is inline.
Next, let’s write the same test with Minitest.
require "test_helper"
class ArticleTest < ActiveSupport::TestCase
setup do
@article = articles(:one)
end
def test_validates_title
@article.title = nil
assert @article.valid?
assert_equal ["can't be blank"], @article.errors[:title]
end
def test_validates_body
@article.body = nil
assert @article.valid?
assert_equal ["can't be blank"], @article.errors[:body]
end
end
Now run the test.
rails test test/models/article_test.rb
# Running:
F
Failure:
ArticleTest#test_validates_body [/Users/williamkennedy/projects/honeybadger/test_minitest/test/models/article_test.rb:17]:
Expected: ["can't be blank"]
Actual: []
rails test test/models/article_test.rb:14
F
Failure:
ArticleTest#test_validates_title [/Users/williamkennedy/projects/honeybadger/test_minitest/test/models/article_test.rb:11]:
Expected: ["can't be blank"]
Actual: []
rails test test/models/article_test.rb:8
Finished in 0.015373s, 130.0982 runs/s, 260.1964 assertions/s.
2 runs, 4 assertions, 2 failures, 0 errors, 0 skips
Straightaway, you will notice one thing. Minitest is just Ruby code, and RSpec is a new language to learn. Although the syntax is similar, RSpec tests can grow in length. There is also more mental overhead when it comes to RSpec, as your team has to define a convention for how tests should be structured.
For me, Minitest is just Ruby, which is a big win. Rails has built a convention into Minitest with the setup method.
The default Minitest file is already set up with the FIRST principle in mind.
UI Testing: RSpec vs. Minitest
Now let’s move on to another pillar of tests. Once again, the Rails default sets up UI tests nested under the test/system folder. However, RSpec has some setup involved, and there is no standard best practice.
A UI test is pretty involved. You write the instructions, such as click here and fill_in, that drive the interaction. They typically use a web driver, such as Selenium, to guide interactions.
They help test JavaScript behavior, recreate user journeys that might have caused errors, and even ensure that specific user flows are airtight.
They are, by their nature, more computationally expensive than regular tests, which is why we may opt for a headless CI tool.
Since Rails introduced system tests, RSpec has benefitted. Previously, we had to set this up manually.
Let’s take an example test. Create a file called spec/system/article_system_spec.rb and add the following code:
require 'rails_helper'
RSpec.describe 'Article', type: :system do
it 'can be created' do
visit '/articles/new'
fill_in 'article[title]', with: 'Hello'
fill_in 'article[body]', with: 'World'
click_button 'Create'
expect(page).to have_content 'Article was successfully created.'
end
end
Running this test will produce what you expect. It will hook into the default web driver, Selenium, at the time of writing. RSpec allows you to change this per test or even all the specs by manually calling the driven_by method.
Here’s how:
require 'rails_helper'
RSpec.describe 'Article', type: :system do
before do
driven_by(:selenium_chrome_headless)
end
it 'should create article' do
visit '/articles/new'
fill_in 'article[title]', with: 'Hello'
fill_in 'article[body]', with: 'World'
click_button 'Create'
expect(page).to have_content 'Article was successfully created.'
end
end
The test will now run with Selenium_chrome_headless instead of Selenium which can speed up your tests.
Since the Capybara library drives the underlying tests, Minitest also has the same syntax.
require "application_system_test_case"
class ArticlesTest < ApplicationSystemTestCase
setup do
@article = articles(:one)
end
test "should create article" do
visit articles_url
click_on "New article"
fill_in "Body", with: @article.body
fill_in "Title", with: @article.title
click_on "Create Article"
assert_text "Article was successfully created"
click_on "Back"
end
end
However, there is a subtle difference in changing your web driver per test.
require "application_system_test_case"
class ArticlesTest < ApplicationSystemTestCase
driven_by :selenium, using: :headless_chrome
setup do
@article = articles(:one)
end
test "should create article" do
visit articles_url
click_on "New article"
fill_in "Body", with: @article.body
fill_in "Title", with: @article.title
click_on "Create Article"
assert_text "Article was successfully created"
click_on "Back"
end
end
You can also change this globally:
# test/application_system_test_case.rb
require "test_helper"
class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
end
Conclusion
Let’s hope this helps you pick between RSpec and Minitest or learn the differences. I have tried to avoid bias as much as possible. There are strengths to both frameworks, but both have the same principles.
Write your tests first, make them pass, and in the long-term, you will have a much happier codebase.
The differences are subtle. With RSpec, you will write a lot more code, and there is a larger ecosystem of plugins to make things easier. However, when the ecosystem is larger and made up of different libraries, dependency trees can occasionally cause difficulty when it comes to upgrading your applications.
One aspect that cannot be ignored is performance.
Minitest is faster than RSpec, but the devil is in the details.
How much faster depends on how you measure. This is difficult to pin down because how you measure matters. Sampling bias(which is when a small sample size is used to come to a conclusion) may favor Minitest over RSpec by a factor of 10x. In other cases, the difference might be just 10%.
There is an interesting library that measures the raw performance of Minitest, RSpec, and Cucumber using Ruby Benchmark and finds the following for Ruby 3:
$ bundle exec ruby ./compare.rb
user system total real
cucumber: 585.035884 22.566803 607.602687 ( 608.237973)
minitest: 18.208514 7.893526 26.102040 ( 26.430622)
rspec: 2406.162561 12.497706 2418.660267 (2418.889164)
test_bench: 29.517226 8.272563 37.789789 ( 38.133189)
Looking at this may cause you think Minitest is the winner by a landslide because it takes less time to run. However, in a real application, it might not be the case. The difference might only be 10%, or there may be no difference at all. Sampling bias is prevalent when it comes to comparing programmer tools due to a myriad of cultural, personal, and other reasons.
If your tests are slow, it may be due to number of other factors, such as database calls, memory, and maybe even network calls.
Choosing between RSpec and Minitest may just come down to personal preference.
Posted on October 5, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.