Introduction

A friend of mine asked me to implement a job-queue. As I was new to Golang at that time I was curious and talked with the friend and also browsed the internet and read the purpose of a job queue.

Eliminate performance bottlenecks

During the research time, I found using JSON is causing a huge impact on performance. The solution was to use gRPC with Protobuf. I used a channel with size with several thousand as the main queue. Since there is limited no of worker machines, I needed to utilize the processor with maximum efficiency to get the jobs done quickly. So each worker machine takes job parameters from the main queue and stores them in their own queue which allows making the room for new requests in the main queue.

These worker machines are equipped with several Goroutines. Each Goroutine takes job parameters from its own worker machine and executes the job.
Even though Goroutines are extremely light-weight thread like programs, using millions of Goroutines won’t result in the optimal utilisation of resources. Please read this article.

My approach to the job-queue implementation
The central server application listens to client requests and put them into the main queue. Several worker machines take job parameters from the main queue and store them on their own queue. Each worker machine is equipped with several small processes whose sole task is to execute the job parameters which are extracted from the queue in the worker machine.

Artifacts

Let the no of worker machines to be m and the no of Goroutines each worker machine runs to be n, on maximum resource utilisation, the applications execute m x n concurrent jobs at any given time.

Source Code

https://gitlab.com/lakshanwd/go-job-queue