What is a good Sidekiq-like job system for node.js?

Question

(Most of the questions asked here about this subject are a bit old, and I was wondering what had change in the Node ecosystem it all those years.)

I'm basically looking to implement a job queue in an app of mine in node.js. I've heard about and seen Sidekiq in action in the Ruby world and how great of a job it does, and was wondering if something similar existed in node.

Workers will be written in Javascript so it doesn't have to be polyglot (it's great if it is, but definitely not a requirement).

A big plus would be an easy way to visualize the jobs currently running, those that failed etc.

What do you guys use? What lib/service do you know about that fits these requirements?

Solution 1

Try https://github.com/taskrabbit/node-resque.

Resque and Sidekiq share the same data structures. In node, getting multiple jobs to process at once is very simple.

Solution 2

Disclaimer: I develop and maintain the node.js module for Faktory.

The author of Sidekiq has created a language-agnostic job queue server backed by redis. It's called Faktory and it implements the best parts of Sidekiq (and more) on the job server side so you can get sidekiq-like job processing in any language (where a client library is available).

The node.js (and TypeScript) client can be found here.

From the README:

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.

You can use this server to distribute jobs to one or hundreds of machines. Jobs can be executed with any language by clients using the Faktory API to fetch a job from a queue.

Basic Features:

• Jobs are represented as JSON hashes.
• Jobs are pushed to and fetched from queues.
• Jobs are reserved with a timeout, 30 min by default.
• Jobs FAIL'd or not ACK'd within the reservation timeout are requeued.
• FAIL'd jobs trigger a retry workflow with exponential backoff.
• Contains a comprehensive Web UI for management and monitoring.

Enterprise and Pro licensing is available for the extra features that you may be familiar with in Sidekiq Pro and Sidekiq Enterprise.

I found myself on this thread because I searched "node.js sidekiq" in search of the same thing and suddenly (embarrassingly) remembered Faktory.

Comments

[stdob--]

Kue - automattic.github.io/kue

[Gokul]

Bull - github.com/OptimalBits/bull

[hsym]

There's a website that list out various libraries for background jobs in various languages: taskqueues.com