Sidekiq worker using a a “custom” queue name gets assigned to “default” queue

queueruby-on-rails-3.2sidekiq

Rails 3.2.9
Ruby 1.9.3dev (2011-09-23 revision 33323) [i686-linux]
sidekiq  2.10.1

If I define a worker like below with a custom queue name

  class BulkEmailWorker
    include Sidekiq::Worker
    # https://github.com/mperham/sidekiq/wiki/Advanced-Options
    sidekiq_options(queue: :bulk_mails, backtrace: true)

    # https://github.com/mperham/sidekiq/wiki/Best-Practices
    # Section: Make your jobs small and simple
    def perform
      # https://github.com/mperham/sidekiq/wiki/Delayed-Extensions
      scheduled_mails = MailTemplate.with_schedule
      scheduled_mails.each do |mail_template|
        BulkMailer.delay.general_mail(mail_template.id)
      end
    end
  end

and I invoke the worker like this:

  BulkEmailWorker.perform_async

it doesn't work (in other words doesn't send email).

Inspecting the data in redis I found following:

  redis 127.0.0.1:6379> keys *
  1) "myapp:stat:processed:2013-04-24"
  2) "myapp:queue:default"
  3) "myapp:stat:processed"
  4) "myapp:queues"

  redis 127.0.0.1:6379> lrange myapp:queue:default -100 100

   1) "{\"retry\":true,\"queue\":\"default\",\"timeout\":30,\"class\":\"Sidekiq::Extensions::DelayedMailer\",\"args\":[\"---\\n- !ruby/class 'BulkMailer'\\n- :general_mail\\n- - 1\\n\"],\"jid\":\"e46693944febf7ae26ec67a0\"}"

redis 127.0.0.1:6379>

As can be seen above the worker is assigned to queue "default" and I guess due to this the worker is unable to process.

I started sidekiq using following command for above scenario:

  bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml -q bulk_mails

However when I remove from my worker following

    sidekiq_options(queue: :bulk_mails, backtrace: true)

and start sidekiq using following command:

  bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml

it works (in other words it sends email successfully).

/config/sidekiq.yml

# https://github.com/mperham/sidekiq/wiki/Logging
# http://stackoverflow.com/questions/15260634/sidekiq-configuration-for-multiple-environments
---
  :verbose: true
  :pidfile: ./tmp/pids/sidekiq.pid
  :logfile: ./log/sidekiq.log
  :concurrency:  25

I have just started using Sidekiq and am novice to it.So please bear with me if I there is some concept which I misunderstood or unaware about.

I need to get rid of the mentioned problem so that my worker gets assigned to the desired queue and also processed successfully.

Thanks,
Jignesh

Best Answer

In addition, note that you can push a job on the queue you want by using Sidekiq::Client.push instead of perform_async:

Sidekiq::Client.push({
  'class' => BulkEmailWorker,
  'queue' => 'bulk_mails',
  'args'  => [ 1, 2 ]
})

Note that the keys (class, queue, args) must be strings, not symbols, or you will get the ArgumentError: Message must include a class and set of arguments.

This is especially useful when you need to push a job on a queue with a variable name (bulk_mails.1, bulk_mails.2, etc.), so you can easily control concurrency by allowing only one job at a time on some queues, thanks to sidekiq-limit_fetch.

Related Topic