✋ Hi this is Suman Sarkar, a web-dev from Kolkata with 5 years of experience in programming and little to none experience with CI/CD. Today I'll talk about how to setup Gitlab CI/CD with self hosted runners.

👉 Things we will cover in this article

1. 😕 What is CI/CD?
2. 👍 Setup a minimal expressjs API with pm2
3. 🧑‍💻 Setup our first ever Gitlab pipeline to install & restart our server whenever an update is pushed on the “dev” branch
4. 🏃 Install self-hosted runners on a linux server
5. 🔒 Register our local runner to Gitlab
6. 🔑 Add environment variables to Gitlab

😕 What is CI/CD?

From my perspective CI/CD or Continuous Integration & Continuous Deployment are processes that you set up for your own convenience so that you don't have to do boring things manually over and over, it is basically automating your workflow when you push an update to your project. Most of us do git pull and then sort of restart the server in order to make the changes into effect, there might be additional steps like building or testing and few other procedures that are specific to your project. I’ll not cover these today, today I’ll only cover how to setup CI/CD for an expressjs application with pm2, Gitlab pipeline and self-hosted runners.

👍 Setup a minimal expressjs API with pm2

We start with creating a directory for our Node JS Express API

mkdir node-cicd-pm2
cd node-cicd-pm2

Then we initialise our project with npm init -y. This creates a package.json file in our project folder with basic information for our project.
Next we add our dependencies by running

npm i –save express dotenv

Lets create our very minimal server by creating our index.js and pasting the below mentioned code.

const express = require('express');
const dotenv = require('dotenv');

const app = express();
dotenv.config();

app.get('', (req, res) => {
    res.status(200).send('Hello World!');
})

app.listen(process.env.PORT, () => {
    console.log(`Server is running on port http://localhost:${process.env.PORT}`);
})

Here, we have required our dependencies express and dotenv then we have added a route that returns 'Hello World!'. We have also added a .env file with only 1 variable.

PORT="3001"

and ecosystem.config.js file with the following content

module.exports = {
    apps: [{
        name: "node-cicd-pm2",
        script: "./index.js"
    }]
}

This will be used later to start our server as a process.

Now, we start our server by running node index.js and visit http://localhost:3001/. It works on my machine! 😈

🏃 Setup our first ever Gitlab pipeline

We start with creating a file specifically named .gitlab-ci.yml. This is an YML file, if you don't like YML, bad news for you, but you can just copy paste and get things done.
Now, paste the following code. I'll explain this in detail.

stages:
  - build_stage
  - deploy_stage

Lets talk about stages, stage are the necessary steps that you can group and describe. We have 2 stages build_stage and deploy_stage. Though we are not building anything here but I like to call it the build stage where we'll install the dependencies. We will cover the deploy stage later.

.base-rules:
  rules:
    - if: '$CI_COMMIT_BRANCH == "dev"'
      when: always
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never

Rules are to describe exactly when your pipeline should run. Here we are specifying that we want to run our pipeline whenever something is pushed onto dev branch by specifying when to always.
$CI_PIPELINE_SOURCE is a special(pre-defined) env. variable provided by Gitlab. It describes the mode our change. These can be the following values push, web, schedule, api, external, chat, webide, merge_request_event, external_pull_request_event, parent_pipeline, trigger, or pipeline. For the same of this article I'll not cover all of them, I am not familiar with most of them anyway.
You can read more about the variables here on Gitlab.

Next up we have caches. The way every stage works is, it cleans or deletes everything a it has produce during its lifetime. In the build stage we will create a node_modules folder which will contain our project's dependencies. When the build_stage is finished we don't want it to be deleted. We want it to passed to the deploy_stage

cache: &global_cache
  key: $CI_COMMIT_REF_SLUG
  policy: pull-push
  paths:
    - node_modules/
    - package-lock.json

We have created a global cache policy here. The policy is pull-push meaning that the stages using this cache policy can pull from global cache and can push to it as well. In order to create new caches with every update, we must provide a slug or an unique identifier. Here we are using $CI_COMMIT_REF_SLUG variable for that. Notice how we are specifying that we only want to cache node_modules directory and package-lock.json since these are the outputs that are generate with npm install.

Let's now define our build_stage

build:
  stage: build_stage
  extends: .base-rules
  script:
    - npm i
  cache:
    <<: *global_cache
    policy: push
  tags:
    - local_runner

The build_stage extends the base_rule so that it will run only when something is pushed on the dev branch.
In this stage we don't want to pull anything from the global-cache, we just want to push the node_modules directory and package-lock.json file in the global-cache. We will cover tags later int this article.

Later we have the deploy_stage

deploy:
  stage: deploy_stage
  extends: .base-rules
  script:
    - "pm2 start ecosystem.config.js"
  cache:
    <<: *global_cache
    policy: pull
  tags:
    - local_runner

In this stage we are pulling the cache from global-cache and then starting our server with pm2 start command. By pulling the cache we get our node_modules directory with our project dependencies.

If you have followed correctly, you should end up with a file with these content

stages:
  - build_stage
  - deploy_stage

.base-rules:
  rules:
    - if: '$CI_COMMIT_BRANCH == "dev"'
      when: always
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never

cache: &global_cache
  key: $CI_COMMIT_REF_SLUG
  policy: pull-push
  paths:
    - node_modules/
    - package-lock.json

build:
  stage: build_stage
  extends: .base-rules
  script:
    - "node --version"
    - npm i
  cache:
    <<: *global_cache
    policy: push
  tags:
    - local_runner

deploy:
  stage: deploy_stage
  extends: .base-rules
  script:
    - "pm2 start ecosystem.config.js"
  cache:
    <<: *global_cache
    policy: pull
  tags:
    - local_runner

💻 Install self-hosted runners on a linux server

A little bit of background on runners, runners are like workers who does something that a computer should do. Like executing any commands or installing your project dependencies. Behind the scene they are docker containers provided by Gitlab. By default Gitlab uses a Ruby container but you can specify your container type. In this article though we will not use Gitlab's runners, we will install our own runner which is an open-source application made by Gitlab and maintained by the dev community. Self hosted runners are completely free so you don't have to worry about money 🤑.

Installing the runner on your server is easy, you just have to run few commands. Visit this page for instruction related to your OS environment. I'm running Ubuntu 20.10 so I'll follow with GNU/Linux Binary guide.. If you are using any debian machine then follow me.. Fire up your terminal and run the following commands..

sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start
sudo gitlab-runner status

Step by step we get the binary, give it executable permissions, create a user called gitlab-runner to run the runners process and then start our gitlab-runner service. The gitlab-runner user is created for security purpose so that it doesn't run as root user. It is generally advised by people who are smarter than me and have was more knowledge about operating systems 😅.
Now, after the last command you should see something like this
Again, it worked on my machine so I'm good! 😅. We are not done with this step though.. We have to login as the gitlab-runner user and install node, npm and pm2. I could not find any reference to what is the default password of gitlab-runner user so I will just reset it using the passwd command.

passwd gitlab-runner

Setup your new password and login as the gitlab-runner user by running su gitlab-runner
For install node I'm using nvm. Just follow the same process mentioned below and you should have everything you need.

curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
source ~/.bashrc

this should install nvm in you machine.
Next, we install node and pm2 globally,

nvm install 16.13.2
npm i -g pm2

🔒 Register our local runner to Gitlab

We are almost done with our setup..
Now, we need to register our runner to Gitlab, to do this go to Setting > CI/CD in your repository and expand the "Runners" section.
At the left side you should see "Specific runners" section.
The token should look something like this "fy7f3BqhVzLq3Mr-xxxx"
In your local machine or wherever you have installed you runner just run

sudo gitlab-runner register

This should prompt you to specify an instance URL. Type https://gitlab.com and press enter.
Then paste the registration token that you found on Gitlab and press enter, next provide a description for your runner
the most important step, providing a tag for your runner or tags. In the .gitlab-ci.yml file I had mention the tags as local_runner so I will put that here. You can add multiple tags separated by comma but that's not mandatory. Tags will identify the runners to do their job. At last choose shell as the executor. The End? Not yet! :'(

🔑 Add environment variables to Gitlab

Now we need to add env variable to Gitlab CI/CD section so that the we can provide a PORT to our application. This is important because .env file is not commited to your version control. We add our env variable PORT under Setting > CI/CD > Variables section and we add the variable as protected. Next, super important - we need to make our dev branch as protected branch. Otherwise it won't fine the variables. You can do this from Settings > Repository > Protected branches section in your repo.

✅ That is it, we are done with our pipeline setup. If everything is done correctly, when you commit a change on your dev branch it should trigger a pipeline with 2 job and you runner should start the pm2 process at 3001 port.

Thanks for reading this article 🧑‍💻 If you face any problems, let me know in the comments down below! 🙂
Happy hacking!