Execute tasks in parallel with a concurrency limit
An interoperable Golang port of the Ruby Que queuing library for PostgreSQL
goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.
goworker is compatible with Resque, so you can push your jobs with Rails and Resque, and consume them with Go in the background.
Machinery is an asynchronous task queue/job queue based on distributed message passing.
Have you ever wanted to write something that is highly concurrent, and performs as many tasks as you will let it, in parallel? Well, look no further, here is a guide on how to do just that, in Go!
Sidekiq compatible background workers in golang.
- reliable queueing for all queues using brpoplpush
- handles retries
- support custom middleware
- customize concurrency per queue
- responds to Unix signals to safely wait for jobs to finish before exiting.
- provides stats on what jobs are currently running
- well tested
NSQ is a successor to simplequeue (part of simplehttp) and as such is designed to (in no particular order):
- support topologies that enable high-availability and eliminate SPOFs
- address the need for stronger message delivery guarantees
- bound the memory footprint of a single process (by persisting some messages to disk)
- greatly simplify configuration requirements for producers and consumers
- provide a straightforward upgrade path
- improve efficiency
Web servers should focus on serving users as quickly as possible. Any non-trivial work that could slow down your user’s experience should be done asynchronously outside of the web process.
Worker queues are commonly used to accomplish this goal. For a more in-depth description of the worker queue architectural pattern, read the Worker Dynos, Background Jobs and Queueing article. Here we’ll demonstrate this pattern using a sample Go application using Que-Go and Heroku Postgres.