Writing worker queues, in Go

Have you ever wanted to write something that is highly concurrent, and performs as many tasks as you will let it, in parallel? Well, look no further, here is a guide on how to do just that, in Go!

go-workers – Sidekiq compatible background workers in golang

Sidekiq compatible background workers in golang.

• reliable queueing for all queues using brpoplpush
• handles retries
• support custom middleware
• customize concurrency per queue
• responds to Unix signals to safely wait for jobs to finish before exiting.
• provides stats on what jobs are currently running
• well tested

NSQ – A realtime distributed messaging platform

NSQ is a successor to simplequeue (part of simplehttp) and as such is designed to (in no particular order):

• support topologies that enable high-availability and eliminate SPOFs
• address the need for stronger message delivery guarantees
• bound the memory footprint of a single process (by persisting some messages to disk)
• greatly simplify configuration requirements for producers and consumers
• provide a straightforward upgrade path
• improve efficiency

Background Jobs with Que-Go

Web servers should focus on serving users as quickly as possible. Any non-trivial work that could slow down your user’s experience should be done asynchronously outside of the web process.

Worker queues are commonly used to accomplish this goal. For a more in-depth description of the worker queue architectural pattern, read the Worker Dynos, Background Jobs and Queueing article. Here we’ll demonstrate this pattern using a sample Go application using Que-Go and Heroku Postgres.