How to implement Worker Pool [Thread Pool] in GoLang ?
3 min readJun 13, 2022
Worker Pool (Or Thread Pool) is something which enables controlled parallel execution of processes. Parallel execution is by running the tasks under multiple Go-Routines and control is done by limiting the number of Go-Routines.
We run a limited number of workers/threads (using a constant defined) always to do processing.
It is achieved by running constant number of Go-Routines in the beginning once the program starts and data is streamlined with the help of buffered channel.
Implementation:
package mainimport (
"fmt"
"strconv"
"sync"
"time"
)const (
threadPoolSize = 5
)var threadPool = make(chan int, threadPoolSize)
var wg = new(sync.WaitGroup)var count int = 0func processEvents(id string, data int) {
fmt.Printf("=====>> Thread %s started\n\n", id)
for i := 1; i < 5; i++ {
time.Sleep(5 * time.Second)
fmt.Printf("=====>> Thread %s in progress. Data : %d\n\n", id, data)
}
fmt.Printf("=====>> Thread %s finished\n\n", id)
}func worker(tId string) {
for data := range threadPool {
processEvents(tId, data)
}
wg.Done()
}func workerPool() {
fmt.Println("WorkerPool started")
defer fmt.Println("WorkerPool finished")
for i := 1; i <= threadPoolSize; i++ {
count++
go worker("Thread-" + strconv.Itoa(count))
}
wg.Done()
}func producer() {
for i := 1; i < 10; i++ {
threadPool <- i
fmt.Println("Produced : ", i)
}
close(threadPool)
}func main() {
start := time.Now()
fmt.Println("Main started")
defer fmt.Println("Main finished. Time taken in seconds : ", time.Since(start).Seconds())
wg.Add(threadPoolSize + 1)
go workerPool()
producer()
wg.Wait()
}
Output :
$ go run main.go Main started
Produced : 1
Produced : 2
Produced : 3
Produced : 4
Produced : 5
WorkerPool started
WorkerPool finished
=====>> Thread Thread-5 startedProduced : 6
=====>> Thread Thread-1 startedProduced : 7
=====>> Thread Thread-2 startedProduced : 8
=====>> Thread Thread-3 startedProduced : 9
=====>> Thread Thread-4 started=====>> Thread Thread-1 in progress. Data : 2=====>> Thread Thread-4 in progress. Data : 5=====>> Thread Thread-5 in progress. Data : 1=====>> Thread Thread-3 in progress. Data : 4=====>> Thread Thread-2 in progress. Data : 3=====>> Thread Thread-2 in progress. Data : 3=====>> Thread Thread-3 in progress. Data : 4=====>> Thread Thread-1 in progress. Data : 2=====>> Thread Thread-4 in progress. Data : 5=====>> Thread Thread-5 in progress. Data : 1=====>> Thread Thread-2 in progress. Data : 3=====>> Thread Thread-5 in progress. Data : 1=====>> Thread Thread-3 in progress. Data : 4=====>> Thread Thread-1 in progress. Data : 2=====>> Thread Thread-4 in progress. Data : 5=====>> Thread Thread-2 in progress. Data : 3=====>> Thread Thread-2 finished=====>> Thread Thread-2 started=====>> Thread Thread-4 in progress. Data : 5=====>> Thread Thread-4 finished=====>> Thread Thread-4 started=====>> Thread Thread-5 in progress. Data : 1=====>> Thread Thread-5 finished=====>> Thread Thread-5 started=====>> Thread Thread-3 in progress. Data : 4=====>> Thread Thread-3 finished=====>> Thread Thread-3 started=====>> Thread Thread-1 in progress. Data : 2=====>> Thread Thread-1 finished=====>> Thread Thread-2 in progress. Data : 6=====>> Thread Thread-3 in progress. Data : 9=====>> Thread Thread-4 in progress. Data : 7=====>> Thread Thread-5 in progress. Data : 8=====>> Thread Thread-2 in progress. Data : 6=====>> Thread Thread-5 in progress. Data : 8=====>> Thread Thread-3 in progress. Data : 9=====>> Thread Thread-4 in progress. Data : 7=====>> Thread Thread-2 in progress. Data : 6=====>> Thread Thread-4 in progress. Data : 7=====>> Thread Thread-5 in progress. Data : 8=====>> Thread Thread-3 in progress. Data : 9=====>> Thread Thread-2 in progress. Data : 6=====>> Thread Thread-2 finished=====>> Thread Thread-3 in progress. Data : 9=====>> Thread Thread-3 finished=====>> Thread Thread-4 in progress. Data : 7=====>> Thread Thread-4 finished=====>> Thread Thread-5 in progress. Data : 8=====>> Thread Thread-5 finishedMain finished. Time taken in seconds : 2.1741e-05
