Run Cucumber tests on Github Actions fast with parallel jobs

arturt

Artur Trzop

Posted on March 26, 2021

Run Cucumber tests on Github Actions fast with parallel jobs

Cucumber employs Behavior-Driven Development (BDD) for testing your application. This type of test is often time-consuming when running in the browser. You will learn how to run Cucumber tests on Github Actions using parallel jobs to execute the test suite much faster.

Alt Text

Github Actions matrix strategy

You can use the Github Actions matrix strategy to run parallel jobs. You will need to divide your Cucumber test files between the parallel jobs in a way that work will be balanced out between the jobs.

It’s not that simple to do because often Cucumber tests can take a different amount of time. One test file can have many test cases, the other can have only a few but very complex ones, etc.

There are often more steps in your CI pipeline like installing dependencies, loading data from the cache and each step can take a different amount of time per parallel job before even Cucumber tests are started. The steps affect the overall CI build speed.

Alt Text

What you would like to achieve is to run parallel jobs in a way that they always finish the execution of Cucumber tests at a similar time. Thanks to that you will avoid lagging jobs that could be a bottleneck in your CI build.

Alt Text

Dynamically split Cucumber tests using Queue Mode

To get optimal CI build execution time you need to ensure the work between parallel jobs is split in such a way as to avoid bottleneck slow job. To achieve that you can split Cucumber test files in a dynamic way between the parallel jobs using Knapsack Pro Queue Mode and knapsack_pro ruby gem.

Knapsack Pro API will take care of coordinating how tests are divided between parallel jobs. On the API side, there is a Queue with a list of your test files and each parallel job on Github Actions is running Cucumber tests via the knapsack_pro Ruby gem. The knapsack_pro gem asks Queue API for a set of test files to run and after it gets executed then the gem asks for another set of test files until the Queue is consumed. This ensures that all parallel jobs finish running tests at a very similar time so that you can avoid bottleneck jobs.

Alt Text

You can learn more about the dynamic test suite split in Queue Mode or check the video below.

Github Actions parallel jobs config for Cucumber

Here is the full Github Actions YAML config example for the Cucumber test suite in a Ruby on Rails project using knapsack_pro gem to run Cucumber tests between parallel jobs.



# .github/workflows/main.yml
name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/master/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.7

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Create DB
        env:
          # use localhost for the host here because we have specified a container for the job.
          # If we were running the job on the VM this would be postgres
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
        run: |
          bin/rails db:prepare

      - name: Run tests
        env:
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:cucumber


Enter fullscreen mode Exit fullscreen mode

Here is the view from Github Actions showing that we run 8 parallel jobs for the CI build.

Alt Text

Summary

I hope you find this example useful. If you would like to learn more about Knapsack Pro please check our homepage and see a list of supported test runners for parallel testing in Ruby, JavaScript, etc.


Originally published at https://docs.knapsackpro.com/2021/cucumber-bdd-testing-using-github-actions-parallel-jobs-to-run-tests-quicker

💖 💪 🙅 🚩
arturt
Artur Trzop

Posted on March 26, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related