How to set retry count for Sidekiq with ActiveJob?

Question

From the Rails API, I found ActiveJob can retry_job interval:

my_job_instance.enqueue
my_job_instance.enqueue wait: 5.minutes
my_job_instance.enqueue queue: :important
my_job_instance.enqueue wait_until: Date.tomorrow.midnight

http://api.rubyonrails.org/classes/ActiveJob/Enqueuing.html

But if I want to set retry count, such as Sidekiq's:

include Sidekiq::Worker
sidekiq_options :retry => 5

https://github.com/mperham/sidekiq/wiki/Error-Handling

How to do in this sample code?

class SiteScrapperJob < ActiveJob::Base
  rescue_from(ErrorLoadingSite) do
    retry_job queue: :low_priority
  end

  def perform(*args)
    # raise ErrorLoadingSite if cannot scrape
  end
end

Now I added this to my job class:

Sidekiq.default_worker_options = { retry: 5 }

But it seems not very good.

Solution 1

As of Sidekiq 6.0.4 you can use sidekiq_options in an ActiveJob to set the retry option.

Solution 2

You also might be interested in this solution which uses serialize and deserialize api to store the number of attempts.

class DeliverWebhookJob < ActiveJob::Base
  def serialize
    super.merge('attempt_number' => (@attempt_number || 0) + 1)
  end

  def deserialize(job_data)
    super
    @attempt_number = job_data['attempt_number']
  end

  rescue_from(ErrorLoadingSite) do |exception|
    retry_job(wait: 10) if @attempt_number < 5
  end

  def perform(*args)
    # raise ErrorLoadingSite if cannot scrape
  end
end

Take it from here.

Solution 3

Since Rails 5.1, there is a built-in way to do this using the retry_on method. It's a general ActiveJob method, so it will work with any queuing backend, not just Sidekiq.

For example, for your specific job you could do:

class SiteScraperJob < ActiveJob::Base
  retry_on ErrorLoadingSite, queue: :low_priority, attempts: 5

  def perform(*args)
    # raise ErrorLoadingSite if cannot scrape
  end
end

You can also set a constant wait interval or an exponential wait strategy, as explained in the docs.

Solution 4

There is a activejob-retry gem which does the job

class SiteScrapperJob < ActiveJob::Base
  include ActiveJob::Retry.new(limit: 5, strategy: :exponential)

  def perform(*args)
    # raise ErrorLoadingSite if cannot scrape
  end
end

---

Another option is to use sidekiq middleware:

First define job_options class-method which will be available in the subclasses:

class ApplicationJob < ActiveJob::Base
  def self.job_options(options)
    @job_options = options
  end

  def self.get_job_options
    @job_options || {}
  end
end

Add middleware which reads job_options from the jobs's class and writes them to the job item for sidekiq:

module Sidekiq
 class JobOptionsMiddleware

   def call(job_wrapper, item, queue, redis_pool)
     job = item['args'][0]['job_class'].constantize

     job.get_job_options
       .each{ |option, value| item[option] = value if item[option].nil? }

     yield
   end

 end

 # in sidekiq initializer

 Sidekiq.configure_client do |config|
   config.client_middleware do |chain|
     chain.add Sidekiq::JobOptionsMiddleware
   end
 end

And finally

 class SiteScrapperJob < ApplicationJob
   job_options retry: 5

   def perform
     # your code
   end
 end

Solution 5

See here the defaults for Sidekiq. The attribute retry "accepts" a boolean value and not a number as you assumed.

From the merge of active_job into Rails this other file one can see that once again retry doesn't accept the number of retries.

What the documentation says then is that per job you can define if the job retries or not.

I also tried to find if the config/sidekiq.yml file can receive this number, and seems like it can't.

Finally,

If you don't fix the bug within 25 retries (about 21 days), Sidekiq will stop retrying and move your job to the Dead Job Queue. You can fix the bug and retry the job manually anytime within the next 6 months using the Web UI.

View more solutions