Unicorn PID is stale on initial deploy

Question

I'm trying to use capistrano and unicorn for the first time in my rails application. I can deploy successfully except unicorn won't start with the following error from stderr:

I, [2013-03-25T16:55:35.877323 #2378]  INFO -- : reloading config_file=/var/www/good/current/config/unicorn.rb E, [2013-03-25T16:55:35.881496 #2378] ERROR -- : error reloading config_file=/var/www/good/current/config/unicorn.rb: Already running on PID:2378 (or pid=/var/www/good/current/tmp/pids/unicorn.pid is stale) (ArgumentError) E, [2013-03-25T16:55:35.881548 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:193:in `pid=' E, [2013-03-25T16:55:35.881579 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/configurator.rb:111:in `block in commit!' E, [2013-03-25T16:55:35.881606 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/configurator.rb:108:in `each' E, [2013-03-25T16:55:35.881645 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/configurator.rb:108:in `commit!' E, [2013-03-25T16:55:35.881674 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:699:in `load_config!' E, [2013-03-25T16:55:35.881703 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:308:in `join' E, [2013-03-25T16:55:35.881731 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/bin/unicorn:126:in `<top (required)>' E, [2013-03-25T16:55:35.881758 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `load' E, [2013-03-25T16:55:35.881786 #2378] ERROR -- : /var/www/good/shared/bundle/ruby/1.9.1/bin/unicorn:23:in `<main>' I, [2013-03-25T16:55:36.233632 #2378]  INFO -- : reaped
#<Process::Status: pid 2392 exit 0> worker=1 I, [2013-03-25T16:55:36.234045 #2378]  INFO -- : reaped
#<Process::Status: pid 2397 exit 0> worker=3 I, [2013-03-25T16:55:36.234560 #2378]  INFO -- : reaped
#<Process::Status: pid 2394 exit 0> worker=2 I, [2013-03-25T16:55:36.336246 #8587]  INFO -- : executing ["/var/www/good/shared/bundle/ruby/1.9.1/bin/unicorn", "-c", "/var/www/good/current/config/unicorn.rb", "-E", "production", "-D", {12=>#<Kgio::UNIXServer:fd 12>, 13=>#<Kgio::TCPServer:fd 13>}] (in /var/www/good/releases/20130325165445) I, [2013-03-25T16:55:36.336510
#8587]  INFO -- : forked child re-executing... I, [2013-03-25T16:55:38.475972 #2378]  INFO -- : reaped
#<Process::Status: pid 2389 exit 0> worker=0 I, [2013-03-25T16:55:38.476109 #2378]  INFO -- : master complete

the key being

error reloading config_file=/var/www/good/current/config/unicorn.rb: Already running on PID:2378

I've included Gemfile, deploy.rb and unicorn.rb

I'd appreciate any help or thoughts. I'm also using RVM and ubuntu

config/deploy.rb

require "rvm/capistrano"
set :rvm_type, :system
set :rvm_install_with_sudo, true

require "bundler/capistrano"
set :bundle_without, [:darwin, :development, :test]

set :application, "Talking GOOD"
set :repository,  "[email protected]:FTW-Development/good.git"
set :scm_user, "[email protected]"
ssh_options[:forward_agent] = true


#set :migrate_target, :current
#set :ssh_options, { :forward_agent => true }

#set :rails_env, "production"
set :deploy_to, "/var/www/good"
#set :normalize_asset_timestamps, false #what?
set :user, "rails"
set :group, "www"
set :use_sudo, false




set :keep_releases, 5
after "deploy:restart", "deploy:cleanup"

require 'capistrano-unicorn'
#after 'deploy:restart', 'unicorn:reload' # app IS NOT preloaded
after 'deploy:restart', 'unicorn:restart'  # app preloaded



#role :web, "your web-server here"                          # Your HTTP server, Apache/etc
#role :app, "your app-server here"                          # This may be the same as your `Web` server
#role :db,  "your primary db-server here", :primary => true # This is where Rails migrations will run
#role :db,  "your slave db-server here"



#set :stages, %(production, staging)
#set :default_stage, "staging"
#require 'capistrano/ext/multistage'

#set(:unicorn_env) { rails_env }
#set(:app_env)     { rails_env }
#role(:web) { domain }
#role(:app) { domain }
#role(:db, :primary => true) { domain }

#set(:deploy_to)    { "/home/#{user}/#{application}/#{fetch :app_env}" }
#set(:current_path) { File.join(deploy_to, current_dir) }

server "good.ftwdev.com", :app, :web, :db, :primary => true

config/unicorn.rb

app_path = "/var/www/good/current"

# Set unicorn options
worker_processes 1
preload_app true
timeout 180
listen "127.0.0.1:9000"

# Spawn unicorn master worker for user apps (group: apps)
user 'rails.www' 

# Fill path to your app
working_directory "/var/www/good/current"

# Should be 'production' by default, otherwise use other env 
rails_env = ENV['RAILS_ENV'] || 'production'

# Log everything to one file
stderr_path "log/unicorn.log"
stdout_path "log/unicorn.log"

# Set master PID location
pid "#{app_path}/tmp/pids/unicorn.pid"

before_fork do |server, worker|
  ActiveRecord::Base.connection.disconnect!

  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  ActiveRecord::Base.establish_connection
end

Gemfile

source 'https://rubygems.org'

gem 'rails', '3.2.8'

# Bundle edge Rails instead:
# gem 'rails', :git => 'git://github.com/rails/rails.git'

gem 'mysql2'

gem 'devise', '~> 2.1.0'
gem 'cancan'

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'
  gem 'bootstrap-sass', '~> 2.2.2.0'

  # See https://github.com/sstephenson/execjs#readme for more supported runtimes
  # gem 'therubyracer', :platforms => :ruby

  gem 'uglifier', '>= 1.0.3'
end


gem 'jquery-rails'
gem 'haml'
gem 'hpricot'
gem 'ruby_parser'
gem 'simple_form'
gem 'high_voltage'
gem 'paperclip', '~> 3.0'
gem 'twitter-bootstrap-rails'


group :production do
  gem 'rb-inotify', '~> 0.9'
  gem 'execjs'
  gem 'therubyracer'
  gem 'unicorn', '~> 4.6'
end

group :development do
  gem 'thin'
  gem 'growl'
  gem 'guard'
  gem 'guard-bundler'
  gem 'guard-livereload'
  gem 'guard-rails'
  gem 'guard-rspec'
  gem 'rack-livereload'
  gem 'html2haml'
  gem 'capistrano'
  gem 'capistrano-unicorn'
  gem 'rvm-capistrano'
  gem 'rb-fsevent', :require => false
  gem 'rb-inotify', '~> 0.9', :require => false
  gem 'rb-fchange', :require => false
end


#gem "rspec-rails", ">= 2.12.1", :group => [:development, :test]
#gem "database_cleaner", ">= 0.9.1", :group => :test
#gem "email_spec", ">= 1.4.0", :group => :test
#gem "cucumber-rails", ">= 1.3.0", :group => :test, :require => false
#gem "launchy", ">= 2.1.2", :group => :test
#gem "capybara", ">= 2.0.2", :group => :test
#gem "factory_girl_rails", ">= 4.1.0", :group => [:development, :test]
#gem "bootstrap-sass", ">= 2.1.1.0"
#gem "devise", ">= 2.2.0"
#gem "cancan", ">= 1.6.8"
#gem "rolify", ">= 3.2.0"
#gem "simple_form", ">= 2.0.4"
#gem "quiet_assets", ">= 1.0.1", :group => :development
#gem "figaro", ">= 0.5.2"  #used for configurations
#gem "better_errors", ">= 0.2.0", :group => :development
#gem "binding_of_caller", ">= 0.6.8", :group => :development




# To use ActiveModel has_secure_password
# gem 'bcrypt-ruby', '~> 3.0.0'

# To use Jbuilder templates for JSON
# gem 'jbuilder'

# Use unicorn as the app server
# gem 'unicorn'

# Deploy with Capistrano
# gem 'capistrano'

# To use debugger
# gem 'debugger'

Solution 1

Your problem is when you're deploying you aren't probably killing the unicorn master pid before you run unicorn again. I think the problem is in your capistrano recipe...

With unicorn we usually save the pid in a shared folder in production environment when we start the server, where you save also the logs, uploads and other sfuff that is independent of each environment.

see this two tasks of one of my deploy.rb:

desc "Start unicorn"
  task :start, :except => { :no_release => true } do
    run "cd #{current_path} ; bundle exec unicorn_rails -c config/unicorn.rb -D"
    run "ps aux | grep unicorn_rails | head -n 1 | awk '{print $2}' > #{deploy_to}/shared/tmp/pids/unicorn.pid"
  end

  desc "Stop unicorn"
  task :stop, :except => { :no_release => true } do
    run "kill -s QUIT `cat  #{deploy_to}/shared/tmp/pids/unicorn.pid`"
  end

so when i stop unicorn i always kill the master pid that is in the shared folder and when i start it, i make a copy of this pid to a shared folder in the production project. In this case, my restart task is calling first the stop and then the start, but this isn't zero down time...

The following link have a recipe with zero down time if want to give a try: https://github.com/railscasts/373-zero-downtime-deployment/blob/master/blog-after/config/recipes/templates/unicorn.rb.erb

Solution 2

I had the same error and the reason was:

while stooping the unicorn server gem 'capistrano3-unicorn' tries to kill unicorn process_id in tmp/pids/unicorn.pid by default but it was at /shared/pids/unicorn.pid.

By changing the path for unicorn.pid in unicorn.rb resolved my error.

Solution 3

There can be three reasons for stale pids: 1) No write permission to the unicorn.pid file. This can be easily detected checking the error logs on starting the unicorn server. 2) In case of remote machines: you have not added unicorn.pid to .gitignore and forcefully overwriting the pids each time with local pid by a unmindful commit. 3) The path of file storing the unicorn.pid in config/unicorn.rb is different from the the path provided in config/deploy.rb in the cap recipe.

Comments

[Itay Grudev]

Your link to the Zero Down Time Recipe is broken.

[tbem]

I updated the link to a zero downtime setup by Ryan Bates