How to put sidekiq into Docker in a rails application?

Question

I am using rails, sidekiq and docker.

My docker-compose.yml file

sidekiq:
  build: .
  command: bundle exec sidekiq -C config/sidekiq.yml
  links:
    - db
    - redis

config/sidekiq.yml file

:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
  - default

After I run docker-compose up, the sidekiq service can not start rightly.

sidekiq_1 | No such file or directory @ rb_sysopen - /testapp/tmp/pids/sidekiq.pid
sidekiq_1 | /usr/local/bundle/gems/sidekiq-3.5.3/lib/sidekiq/cli.rb:365:in `initialize'
sidekiq_1 | /usr/local/bundle/gems/sidekiq-3.5.3/lib/sidekiq/cli.rb:365:in `open'
sidekiq_1 | /usr/local/bundle/gems/sidekiq-3.5.3/lib/sidekiq/cli.rb:365:in `write_pid'
sidekiq_1 | /usr/local/bundle/gems/sidekiq-3.5.3/lib/sidekiq/cli.rb:42:in `parse'
sidekiq_1 | /usr/local/bundle/gems/sidekiq-3.5.3/bin/sidekiq:12:in `<top (required)>'
sidekiq_1 | /usr/local/bundle/bin/sidekiq:16:in `load'
sidekiq_1 | /usr/local/bundle/bin/sidekiq:16:in `<main>'
testapp_sidekiq_1 exited with code 1

Solution 1

Try to mount the volumes. Your docker-compose file should look like this (with PosgtreSQL as database) :

web:
  build: .
  volumes:
    - .:/myapp
  links:
    - db
    - redis
  ports:
    - "3000:3000"
  command: bundle exec rails server -b 0.0.0.0
sidekiq:
  build: .
  volumes:
    - .:/myapp
  links:
    - db
    - redis
  command: bundle exec sidekiq
db:
  image: postgres
redis:
  image: redis

Solution 2

A little bit more details in general on how to put Sidekiq into Docker in a Rails application. And for your reference, all the code is available on a GitHub repository.

Configure a Redis container

Sidekiq depends on Redis. So first of all, you need a Redis container to run alongside.

In your docker-compose.yml, add (as an example):

redis:
  image: redis:4.0-alpine

Dockerize Sidekiq

Share the same Dockerfile with your Rails app:

FROM ruby:2.3.3
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs
RUN mkdir /myapp
WORKDIR /myapp
ADD Gemfile /myapp/Gemfile
ADD Gemfile.lock /myapp/Gemfile.lock
RUN bundle install
ADD . /myapp

Update your docker-compose.yml

sidekiq:
  build: .
  command: bundle exec sidekiq
  depends_on:
    - redis
  volumes:
    - .:/myapp
  env_file:
    - .env

The environment file .env looks like this:

JOB_WORKER_URL=redis://redis:6379/0

Also in your docker-compose.yml, add sidekiq to your Rails application's dependency list:

web:
  build: .
  command: bundle exec rails s -p 3000 -b '0.0.0.0'
  volumes:
    - .:/myapp
  ports:
    - "3000:3000"
  depends_on:
    - db
    - sidekiq
  env_file:
    - .env

Add to your Gemfile

gem 'sidekiq'
gem 'sidekiq-scheduler'
gem 'sidekiq-unique-jobs'
gem 'sinatra', require: nil

The sinatra gem is needed for the Sidekiq web UI (the dashboard shown below)

Configure Sidekiq

Add file config/initializers/sidekiq.rb:

sidekiq_config = { url: ENV['JOB_WORKER_URL'] }

Sidekiq.configure_server do |config|
  config.redis = sidekiq_config
end

Sidekiq.configure_client do |config|
  config.redis = sidekiq_config
end

Add a Sidekiq worker

Under directory app/workers/, add a file my_worker.rb:

class MyWorker
  include Sidekiq::Worker
  def perform(who, message)
    logger.info "Message from #{who} is #{message}"
  end
end

That's it. Now you can submit a job in Rails, e.g., in a controller.

MyWorker.perform_async(who, message)

And the worker will pick up the job, and output a message in the log file.

Build and run with docker compose

Once everything is in place, you can build docker images and run your app with docker compose:

docker-compose build
docker-compose up

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                    NAMES
c0515ac60a8b        hellosidekiq_web       "bundle exec rails..."   23 minutes ago      Up 23 minutes       0.0.0.0:3000->3000/tcp   hellosidekiq_web_1
080e33963e3a        hellosidekiq_sidekiq   "bundle exec sidekiq"    23 minutes ago      Up 23 minutes                                hellosidekiq_sidekiq_1
80d1c03f0573        redis:4.0-alpine       "docker-entrypoint..."   4 days ago          Up 23 minutes       6379/tcp                 hellosidekiq_redis_1
5915869772e4        postgres               "docker-entrypoint..."   4 days ago          Up 23 minutes       5432/tcp                 hellosidekiq_db_1

Test

Now open the following URL to submit a job:

http://localhost:3000/job/submit/John/Prepare%20ye%20the%20way

And in the log file, you will see something like this:

sidekiq_1  | 2017-11-13T17:08:45.876Z 1 TID-qw47g MyWorker JID-b7b6d39b0d5193cd01e97cb1 INFO: Message from John is Prepare ye the way

Sidekiq dashboard

If you would like to use the Sidekiq dashboard, as below

You can add the route to your routes.rb

require 'sidekiq/web'
require 'sidekiq-scheduler/web'

Rails.application.routes.draw do
  # For details on the DSL available within this file, see http://guides.rubyonrails.org/routing.html
  mount Sidekiq::Web => '/sidekiq'

  get 'job/submit/:who/:message', to: 'job#submit'
end

Hope it helps.

By the way, if you would like to find out how to dockerize your Rails application using docker compose, refer to docker compose documentation.

Comments

[scho]

Thank you. Can this method load the sidekiq config file -C config/sidekiq.yml?

[Ben]

it seems handy to specify the REDIS_URL also

[lcharbon]

Why do both the sidekiq and ` web` containers need to be mapped to the same external directory? Why can't they each use a copy of /myapp inside their respective containers? I get the above error if I don't map the volumes.

[lcharbon]

Resolve the above issue and was able to use a copy from inside the container by adding: RUN mkdir -p /usr/src/soloid-api/tmp/pids To my dockerfile. Unlike rails Sidekiq doesn't create a pids folder and gets confused.

[yaswant singh]

in this case sidekiq and web both create 2 container. How i can configure in one container both services. Thanks

[Kaka Ruto]

You are the absolute best, @Yuci!

[Ian Vaughan]

I dont think sidekiq needs the db link?