Introduction

Real-time data tracking is essential for monitoring transportation of goods and services. I used Golang to build an application that simulates GPS movement, ingest the time-series data in QuestDB and there’s a REST API that exposes the real-time data and metrics. We will leverage Docker to integrate the backend, QuestDB as the time-series database and a data-source for Grafana visualization and the deployment process will be automated using Jenkins.

Tech Stack

• Golang for Backend
• QuestDB for Time-Series Database
• Grafana for Data Visualization
• Jenkins for CI/CD
• Docker for Containerization
• Prometheus for Scraping and Monitoring

Prerequisites

• 2 Ubuntu 22.04 Servers
  • Jenkins-Server
  • Application
• Ensure port 8080 is open on Jenkins-Server
• Install Docker on both servers

Setting up Jenkins

The following are taken to setup the Jenkins server:

1. SSH into the server and create a script called jenkins.sh and paste the following code:

#!/bin/bash
sudo apt update
sudo apt install openjdk-17-jre -y
sudo apt install openjdk-11-jdk -y
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian/jenkins.io-2023.key

echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update
sudo apt install jenkins -y
sudo systemctl start jenkins
sudo systemctl enable jenkins

1. Make the script executable and run it:

sudo chmod +x jenkins.sh && ./jenkins.sh

1. Jenkins comes with a default admin password. To find the password, run the following command:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

1. To access Jenkins server, open your browser and paste http:<jenkins_server_ip>:8080.

2. Paste the password and click on Continue.

3. Install suggested plugins and create an admin user.

1. Once Jenkins is installed, you can access it by visiting http://<jenkins_server_ip>:8080 in your web browser.

1. Some plugins are required for Jenkins to function properly. To install them, click on Manage Jenkins in the left sidebar, then click on Plugins.

2. Click on Available Plugins and search for the following plugins:

   • GitHub Integration Plugin
   • Blue Ocean Plugin
   • Docker Plugin

3. Click on each plugin, install and do not select Restart Jenkins when installation is complete and no jobs are running.

Setting up Jenkins Pipeline

To create a Jenkins pipeline, follow these steps:

1. On the Jenkins dashboard, click on New Item in the left sidebar.

1. Enter a name for your pipeline, select Pipeline and click on OK.

1. Give the pipeline a description, select Discard old builds and choose 2 the number of builds to keep ensuring server memory usage is optimized.

1. Select GitHub project and paste the repository URL of the project.

https://github.com/ekedonald/stackup-testing.git

1. Select GitHub hook trigger for GITScm polling, this needs to be enabled for Jenkins to automatically trigger the pipeline when changes are pushed to the repository.

1. In the pipeline configuration page, click on Pipeline and select Pipeline script from SCM.

2. Fill in the following fields:

   • SCM: Git
   • Repository URL: https://github.com/ekedonald/stackup-testing.git
   • Branch Specifier: */main
   • Script Path: Jenkinsfile

1. Click on Save to save the pipeline configuration.

Configure Webhook on GitHub

To configure the webhook on GitHub, follow these steps:

1. Go to the repository settings page on GitHub and click on Webhooks in the left sidebar.

2. Click on Add webhook.

3. Fill in the following fields and click on Add webhook:

   • Payload URL: http://<jenkins_server_ip>:8080/github-webhook/
   • Content type: application/json

Configure Secrets and Secrets on Jenkins

To configure secrets and secrets on Jenkins, follow these steps:

1. On the Jenkins dashboard, click on Manage Jenkins and Add Credentials.

2. On stores scoped to jenkins, click on global and Add Credentials.

1. Create a new credential for the env with the following details:
   • Kind: Secret file
   • File: upload or drag the env file
   • ID: env-file-secret

1. Create new credentials for the private key with the following details:
   • Kind: Username with password
   • ID: ssh-pem-key
   • Username: remote server username, treat username as secret
   • Private key: upload or drag the private key

1. Create a new credential for the remote-user with the following details:
   • Kind: Secret text
   • Secret: remote server username
   • ID: remote-user

1. Create a new credential for the remote-host with the following details:
   • Kind: Secret text
   • Secret: remote server password
   • ID: remote-host

Configure your Jenkinsfile

In the github repository, create a file called Jenkinsfile and paste the following code:

pipeline {
    agent any

    environment {
        DOCKER_IMAGE = 'delivery-tracker'
        DOCKER_TAG = "${BUILD_NUMBER}"
        PEM_PATH = "/tmp/deploy-key-${BUILD_NUMBER}.pem"
        TEMP_DIR = "/tmp/deployment-${BUILD_NUMBER}"
        GIT_REPO = 'https://github.com/ekedonald/stackup-testing.git'
        REMOTE_DIR = '/home/ubuntu/stackup-testing'
    }

    stages {
        stage('Setup SSH') {
            steps {
                script {
                    withCredentials([
                        string(credentialsId: 'remote-user', variable: 'REMOTE_USER'),
                        string(credentialsId: 'remote-host', variable: 'REMOTE_HOST'),
                        sshUserPrivateKey(credentialsId: 'ssh-pem-key', keyFileVariable: 'SSH_KEY')
                    ]) {
                        sh """
                            cp "\$SSH_KEY" ${PEM_PATH}
                            chmod 600 ${PEM_PATH}
                            ssh-keyscan -H \$REMOTE_HOST >> /tmp/known_hosts_${BUILD_NUMBER}
                        """
                    }
                }
            }
        }

        stage('Create .env File') {
            steps {
                script {
                    withCredentials([file(credentialsId: 'env-file-secrets', variable: 'ENV_FILE')]) {
                        sh 'cp $ENV_FILE .env'
                    }
                }
            }
        }

        stage('Build Docker Image') {
            steps {
                script {
                    sh "docker build -t ${DOCKER_IMAGE}:${DOCKER_TAG} ."
                    sh "docker save ${DOCKER_IMAGE}:${DOCKER_TAG} > ${DOCKER_IMAGE}.tar"
                }
            }
        }

        stage('Transfer and Deploy') {
            steps {
                script {
                    withCredentials([
                        string(credentialsId: 'remote-user', variable: 'REMOTE_USER'),
                        string(credentialsId: 'remote-host', variable: 'REMOTE_HOST')
                    ]) {
                        sh """
                            ssh -i ${PEM_PATH} -o UserKnownHostsFile=/tmp/known_hosts_${BUILD_NUMBER} \
                                \$REMOTE_USER@\$REMOTE_HOST '\
                                mkdir -p ${TEMP_DIR}'

                            scp -i ${PEM_PATH} -o UserKnownHostsFile=/tmp/known_hosts_${BUILD_NUMBER} \
                                ${DOCKER_IMAGE}.tar \$REMOTE_USER@\$REMOTE_HOST:${TEMP_DIR}/

                            scp -i ${PEM_PATH} -o UserKnownHostsFile=/tmp/known_hosts_${BUILD_NUMBER} \
                                .env \$REMOTE_USER@\$REMOTE_HOST:${TEMP_DIR}/

                            ssh -i ${PEM_PATH} -o UserKnownHostsFile=/tmp/known_hosts_${BUILD_NUMBER} \
                                \$REMOTE_USER@\$REMOTE_HOST '\
                                git clone ${GIT_REPO} ${REMOTE_DIR} && \
                                cp ${TEMP_DIR}/.env ${REMOTE_DIR}/ && \
                                docker load < ${TEMP_DIR}/${DOCKER_IMAGE}.tar && \
                                rm -rf ${TEMP_DIR} && \
                                cd ${REMOTE_DIR} && \
                                sed -i "s|build: .|image: ${DOCKER_IMAGE}:${DOCKER_TAG}|g" compose.yaml && \
                                docker compose up -d'
                        """
                    }
                }
            }
        }
    }

    post {
        always {
            sh """
                rm -f ${PEM_PATH}
                rm -f /tmp/known_hosts_${BUILD_NUMBER}
                rm -f ${DOCKER_IMAGE}.tar
                rm -f .env
            """
            cleanWs()
        }
        success {
            echo 'Deployment completed successfully!'
        }
        failure {
            echo 'Deployment failed!'
        }
    }
}

The pipeline above is configured to:

• Build a Docker image for the application delivery-tracker
• Transfer the Docker image to the remote server
• Deploy the application to the remote server
• Clean up the temporary files and resources

Trigger the Jenkins Pipeline from GitHub

• Update a file in your GitHub repository and it will automatically trigger the Jenkins pipeline.
• Go to the Jenkins dashboard, click on the build number dropdown and select Pipeline Overview to view the build logs.
• The pipeline build is successfully and you should be able to access the application at http://:3000 to access Grafana.

Grafana will be integrated with questdb as a datasource and we can analyse metrics such as Total Distance Covered and identify Top Performing Drivers.