_{^{Photo by Ruslan Bardash on Unsplash}}

Scenario

In building and testing a monorepo in CI, there are problems regarding splitting the tasks in order to lower the time of the CI run.

NX Solution

Nx has a feature - distributed task execution to run tasks parallel in CI that helps speed up building and testing a monorepo.

Unlike manually splitting the build and tests into different jobs, in NX distributed execution the tasks are distributed by a automated manager. The user decided the number of workers to give and the manager utilizes them.

TLDR

CI.yaml - Main Section

name: CI
on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true

CI.yaml - Job - Main

jobs:
  main:
    runs-on: ubuntu-latest
    if: ${{ github.event_name != 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'
      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2

      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

CI.yaml - Job - PR

  pr:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'pull_request' }}
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - run: pnpm exec nx-cloud start-ci-run
      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3
      - run: pnpm exec nx affected --target=lint --parallel --max-parallel=2
      - run: pnpm exec nx affected --target=test --parallel --max-parallel=2
      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

CI.yaml - Job - Agents

  agents:
    runs-on: ubuntu-latest
    name: Agent - General
    timeout-minutes: 15
    strategy:
      matrix:
        agent: [1, 2]
    steps:
      - uses: actions/checkout@v2'

      - uses: pnpm/action-setup@v2.0.1
        with:
          version: 6.22.2

      - uses: actions/setup-node@v2
        with:
          node-version: '17'
          cache: 'pnpm'

      - run: pnpm install
      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent

Solution Walkthrough

Activating the Distributed Execution Manager

In order for the NX commands to know this they should use the manager for work dispatching, they have to have the environment variable NX_CLOUD_DISTRIBUTED_EXECUTION to true.

We put this variable environment wide for all jobs and steps.

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true

If we want to disable the usage of distribute tasks for a specific NX command we can set this variable to false temporarily for a specific step for instance.

      - name: Publish artifacts
        run: pnpm exec nx affected --target=custom-publish --parallel=2 --commitid=$GITHUB_SHA
        env:
          NX_CLOUD_DISTRIBUTED_EXECUTION: false

NX distributed execution can only run commands that are specified as cachable in the NX comnfig. Commands that are not specified as cachable gives error.

Checking out Branch

To be able to determine which packages were changed - meaning needing build and test - on the main and pr branches, we want to retrieve all the repo commits. Otherwise, NX Affected command will not be able to compare commits regarding packages changes.

We do this by specifying to download all commits in the checkout action.

fetch-depth: 0 - Specify to download all commits

For determine changes of pr branches, we want to checkout not the special branch Github creates which is the merge between the PR branch and the base it is against, but the PR branch head only.

This is done by specifying ref: ${{ github.event.pull_request.head.ref }}

The event.pull_request.head.ref contains the branch name which is the reference to the head commit

The tests and builds themselves will be performed in the agents which will checkout the merged PR commit as usual.

Job - main

      - uses: actions/checkout@v2
        name: Checkout [main]
        with:
          fetch-depth: 0

Job - pr

      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

Helping NX Affect Command Comparison

For PR's we want to compare the head of the PR branch against the base of the PR.

For main branch commits, the desire is to find changes against the latest commit that was succesful.

NX provide an action utility that calculate those head and base commits and set those to environment variables - NX_BASE and NX_HEAD - used by the NX Affected command.

      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2

Starting the Distributed Execution Manager

To start the manager daemon, we execute nx-cloud utility with start-ci-run command.

      - run: pnpm exec nx-cloud start-ci-run

Dispatching a Distributed Task

The NX Affect command run without customization.

The parallel/max-parallel does not specify the number of workers/agents but specify the number of processes to run parallel on each agent and on the manager.

      - run: pnpm exec nx affected --target=build --parallel --max-parallel=3

Cleanup Agent After Success and Failure

After the job is done or after a build error, we want to signal the agents to shutdown and not keep waiting for future tasks to execute.

This is done by activating nx-cloud stop-all-agents.

In order to do this step even on error, the step needs to specify that it will run always and not just on success.

if: ${{ always() }} - Do the step always

      - name: Clean Agents
        if: ${{ always() }}
        run: pnpm exec nx-cloud stop-all-agents

Agent Job Configuration

The Agent has one shared configuration. To parallel and multiple the number of agents node, we use the matrix feature of the job.

It will create parallel worker nodes, each with different agent name variable value.

    strategy:
      matrix:
        agent: [1, 2]

Rest of the agent job configuration is standard. Checking out the repo merged commit, setting up nodejs, and starting up the agent server that listen for the tasks manager task dispatches.

      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent

Conclusion

For monorepo scalable testing and building, checkout NX and NX Cloud

It provide an interesting solution for efficient parallel task execution to speedup monorepos CI workflows.