Continuous Integration/Deployment for Rails on Gitlab

In this blog post, we will through the necessary steps to setup Gitlab in order
to run Rails build, tests & deployment if everything will be okay.
I will put a particular attention on rails system test and how to make them works.

We will use Heroku to deploy our staging App.

What will we achieve ?

The build Stage

The build will contain:

• Installation of dependencies
• Database setup
• Precompile of assets (assets & webpacker)

The Test Stage

The Integration tests

In this stage, we will run all our integration tests, which basically turn to
run:
bundle exec rails test

The system tests

This is the most exciting and important part to have in our CI.
The system tests are very useful in term of testing complex UI requiring a massive use
of Javascript (React of Vue app) and interacting with external services like Google
Map Places.

The system test will mimic a regular user by clicking and filling inputs like a regular user on our App.
The main command executed in this stage is bundle exec rails test:system.

The interacting fact in this case is the use of container to embed the Selenium
Chrome browser to run real browser to fetch and tests our frontend.

The deploy Stage

This is an easy step, we will deploy our application to the staging environment.

The GITLAB-CI

Gitlab offer to everyone ( and we should be grateful to them for all the work
that have be done) a recipe that define how the code will be tested / deployed
and all the services needed for these tasks.
All the instructions are stored in .gitlab-ci present in the root of our repo.

This offer us a centralized and an easy way to manage our source code and
our continues integration for FREE.

How does it works

The CI follow these simple steps:

1. Booting one or several containers aka services that you have specified in the .gitlab-ci
2. Copy your repo in the main container.
3. Run all scripts wanted in it

Use the cache to speedup the CI

Gitlab allows us to cache folders and files and use them for the next jobs.
No need to recompile all dependencies or even download them.
In our case, caching all the gems and node_modules will save us several minutes.

Use artifacts to debug our tests

When the system tests fails, the test will save the screenshots in a temp folder.

The artifacts make possible for us to save those files and tie them to the job.

This will help us a lot when we want to debug a failing system tests.

Let's do it

1. The build

Prepare the build container

The build will be executed in a container, so we should have a container with
all the dependencies needed bundled inside.

For the modern rails app we should include:

• Ruby
• Node + Yarn
• Some system libraries

Here is the dockerfile

FROM ruby:2.4.3

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qqy && apt-get install  -qqyy yarn nodejs postgresql postgresql-contrib libpq-dev cmake

RUN rm -rf /var/lib/apt/lists/*

Easy yay !

Build the container

docker build .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM ruby:2.6.5
2.6.5: Pulling from library/ruby
16ea0e8c8879: Pull complete
50024b0106d5: Pull complete
ff95660c6937: Pull complete
9c7d0e5c0bc2: Pull complete
29c4fb388fdf: Pull complete
069ad1aadbe0: Pull complete
e7188792d9dd: Pull complete
bae7e74440d1: Pull complete
Digest: sha256:2285f291f222e1b53d22449cc52bad2112f519bcce60248ea1c4d5e8f14c7c04
Status: Downloaded newer image for ruby:2.6.5
 ---> 2ff4e698f315
Step 2/6 : RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
 ---> Running in abb67e50af3e
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
Removing intermediate container abb67e50af3e
 ---> 461e2dd2134d
Step 3/6 : RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
 ---> Running in 414f508a391c

## Installing the NodeSource Node.js 8.x LTS Carbon repo...

.....
Processing triggers for libc-bin (2.28-10) ...
Removing intermediate container af1183021a8d
 ---> 603cab5f6952
Step 6/6 : RUN rm -rf /var/lib/apt/lists/*
 ---> Running in 53c5950a25c1
Removing intermediate container 53c5950a25c1
 ---> 42b50699301e
Successfully built 42b50699301e

Tag it

docker tag 42b50699301e registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1

Now, we should publish this container to enable GitlabCI to use it.

Gitlab provide for us a container registry ! for free again !

So we just need to push this container in the project registry.

First, you should login to gitlab registry

docker login registry.gitlab.com
# use your gitlab credential

and PUSH

docker push registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1 # v1 is my version tag

If you have ADSL internet connection with a poor uploading speed, you can go
take a nap ;)

Once the push finishes, we are good to go to the next step.

The build script

This is the main build part in the gitlab-ci file

image: "registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1"

variables:
  LC_ALL: C.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  RAILS_ENV: "test"
  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

# cache gems and node_modules for next usage
.default-cache: &default-cache
  cache:
    untracked: true
    key: my-project-key-5.2
    paths:
      - node_modules/
      - vendor/
      - public/

build:
  <<: *default-cache
  services:
    - postgres:latest
  stage: build
  script:
  - ruby -v
  - node -v
  - yarn --version
  - which ruby
  - gem install bundler  --no-ri --no-rdoc
  - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
  - yarn install
  - cp config/database.gitlab config/database.yml
  - RAILS_ENV=test bundle exec rake db:create db:schema:load
  - RAILS_ENV=test bundle exec rails assets:precompile

So we are using the previously created image to host the build.

We should add to the project a config/database.gitlab to replace the original
database config and use custom host and credential to connect to postgres
container booted by the GitlabCI.

  services:
    - postgres:latest

Gitlab when reading this line will bootup a database container (postgress) and
will use the variables defined before to setup the database

  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

The config/database.gitlab will tell our rails app how to connect to the
database, so before the app boots, the database.yml will be replaced by the
custom one.

test:
  adapter: postgresql
  encoding: unicode
  pool: 5
  timeout: 5000
  host: postgres
  username: runner
  password: ""
  database: test_db

2. The Integration Tests script

No need more explanation for this

integration_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - bundle install --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test

3. The System Tests script

The infrastructure to make possible the system test is quite interesting.

To run the test we should start a browser (in a container) and fetch the page
from the rails server (from an other container).

system_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
    - selenium/standalone-chrome:latest
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - export selenium_remote_url="http://selenium__standalone-chrome:4444/wd/hub/"
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test:system
  artifacts:
    when: on_failure
    paths:
      - tmp/screenshots/

We should tell to capybara to use the right IP instead of localhost, because here we have the browser
and the server in two different containers.

In the environment/test.rb, add these lines

  net = Socket.ip_address_list.detect{|addr| addr.ipv4_private? }
  ip = net.nil? ? 'localhost' : net.ip_address
  config.domain = ip
  config.action_mailer.default_url_options = { :host => config.domain }

  Capybara.server_port = 8200
  Capybara.server_host = ip

and we should tell the system test where to find the chrome driver to control the browser, update application_system_test_case.rb

require "test_helper"
require "socket"


def prepare_options
  driver_options = {
    desired_capabilities: {
      chromeOptions: {
        args: %w[headless disable-gpu disable-dev-shm-usage] # preserve memory & cpu consumption
      }
    }
  }

  driver_options[:url] = ENV['selenium_remote_url'] if ENV['selenium_remote_url']

  driver_options
end

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400],
            options: prepare_options
end

The rails system test will take screenshots and save them to tmp/screenshots

{:class="img-responsive"}

As you can see, the screenshots are stored and attached to job, Neat!

4. The Staging deployment

This will deploy our code if build and tests stages succeed.

deploy_staging:
  stage: deploy
  variables:
    HEROKU_APP_NAME: YOUR_HEROKU_APP_NAME
  dependencies:
    - integration_test
    - system_test
  only:
    - master
  script:
    - gem install dpl
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_API_KEY

The HEROKU_API_KEY is stored in a safe place in the settings of the project

For more information about this, go to Gitlab variables documentation

Conclusion

Gitlab is an amazing project and provide a very nice spot where everything is
well integrated, the coding experience is enhanced.

Finally, let's hope that the migration to Google compute engine will provide a
better robustness to the project and less issues.

Longue vie à Gitlab !!

Cheers!

Here is the complete Gitlab CI file