Have you ever wanted to write something that is highly concurrent, and performs as many tasks as you will let it, in parallel? Well, look no further, here is a guide on how to do just that, in Go!
Sidekiq compatible background workers in golang.
- reliable queueing for all queues using brpoplpush
- handles retries
- support custom middleware
- customize concurrency per queue
- responds to Unix signals to safely wait for jobs to finish before exiting.
- provides stats on what jobs are currently running
- well tested
NSQ is a successor to simplequeue (part of simplehttp) and as such is designed to (in no particular order):
- support topologies that enable high-availability and eliminate SPOFs
- address the need for stronger message delivery guarantees
- bound the memory footprint of a single process (by persisting some messages to disk)
- greatly simplify configuration requirements for producers and consumers
- provide a straightforward upgrade path
- improve efficiency
Web servers should focus on serving users as quickly as possible. Any non-trivial work that could slow down your user’s experience should be done asynchronously outside of the web process.
Worker queues are commonly used to accomplish this goal. For a more in-depth description of the worker queue architectural pattern, read the Worker Dynos, Background Jobs and Queueing article. Here we’ll demonstrate this pattern using a sample Go application using Que-Go and Heroku Postgres.