A friend of mine asked me to implement a job-queue. As I was new to Golang at that time I was curious and talked with the friend and also browsed the internet and read the purpose of a job queue.
Eliminate performance bottlenecks
During the research time, I found using
JSON is causing a huge impact on performance. The solution was to use gRPC with Protobuf. I used a channel with size with several thousand as the main queue. Since there is limited no of worker machines, I needed to utilize the processor with maximum efficiency to get the jobs done quickly. So each worker machine takes job parameters from the main queue and stores them in their own queue which allows making the room for new requests in the main queue.
My approach to the job-queue implementation
The central server application listens to client requests and put them into the main queue. Several worker machines take job parameters from the main queue and store them on their own queue. Each worker machine is equipped with several small processes whose sole task is to execute the job parameters which are extracted from the queue in the worker machine.
Let the no of worker machines to be
m and the no of Goroutines each worker machine runs to be
n, on maximum resource utilization, the applications execute
n concurrent jobs at any given time.