Occasional Postgres error on Heroku: could not translate host name "<pg URL>" to address: Name or service not known (PG::Error)

Question

I'm using the shared postgres (dev / free) plan right now on Heroku and I see this error in my logs a decent amount. Connecting to the PG database does usually work though, so I'm not sure if this is something that's within my control. The Heroku Status page does not list any downtime for the shared databases when these occur.

Using Rails 3.1.1 and the Unicorn web server.

Error: could not translate host name "pg60.sharedpg.heroku.com (http://pg60.sharedpg.heroku.com)" to address: Name or service not known (PG::Error)

My Unicorn config file (after looking around at the recommended settings for Unicorn in Heroku):

worker_processes 3 # amount of unicorn workers to spin up
timeout 30         # restarts workers that hang for 30 seconds
preload_app true

before_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection_handler.clear_all_connections!
    ActiveRecord::Base.connection.disconnect!
    Rails.logger.info('Disconnected from ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
    ActiveRecord::Base.connection_handler.verify_active_connections!
    Rails.logger.info('Connected to ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

Trace:

 328 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `initialize': could not translate host name "pg60.sharedpg.heroku.com (http://pg60.sharedpg.heroku.com)" to address: Name or service not known (PG::Error)
219 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `new'
223 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `connect'
225 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:301:in `initialize'
217 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `new'
235 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `postgresql_connection'

328 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `initialize': could not translate host name "pg60.sharedpg.heroku.com (http://pg60.sharedpg.heroku.com)" to address: Name or service not known (PG::Error)
219 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `new'
223 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:1032:in `connect'
225 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:301:in `initialize'
217 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `new'
235 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `postgresql_connection'
235 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:304:in `new_connection'
244 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:323:in `checkout_new_connection'
249 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block (2 levels) in checkout'
225 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:261:in `loop'
238 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:261:in `block in checkout'
153 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
229 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:260:in `checkout'
231 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:162:in `connection'
240 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:409:in `retrieve_connection'
249 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_specification.rb:107:in `retrieve_connection'
239 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/connection_adapters/abstract/connection_specification.rb:89:in `connection'
194 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.1.1/lib/active_record/base.rb:703:in `table_exists?'
128 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/app.rb:38:in `<top (required)>'
116 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from config.ru:9 (http://config.ru:9):in `require'
124 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from config.ru:9 (http://config.ru:9):in `block in <main>'
179 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/builder.rb:51:in `instance_eval'
176 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/builder.rb:51:in `initialize'
112 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from config.ru:1 (http://config.ru:1):in `new'
115 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from config.ru:1 (http://config.ru:1):in `<main>'
168 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/lib/unicorn.rb:44:in `eval'
180 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/lib/unicorn.rb:44:in `block in builder'
181 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/lib/unicorn/http_server.rb:691:in `call'
187 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/lib/unicorn/http_server.rb:691:in `build_app!'
182 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/lib/unicorn/http_server.rb:136:in `start'
178 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/gems/unicorn-4.2.0/bin/unicorn:121:in `<top (required)>'
146 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/bin/unicorn:19:in `load'
148 <13>1 2012-08-27T21:36:37+00:00 d.4122435c-cb44-421a-9a7d-a68bb10457d6 app web.1 (http://web.1) - - from /app/vendor/bundle/ruby/1.9.1/bin/unicorn:19:in `<main>' 

Solution 1

That's a DNS error. It's exceedingly unlikely to be anything to do with your code. If both your Rails app and your DB are within Heroku, contact Heroku support.

If your Rails app is running outside the Heroku cloud (as a bit of searching for the term "Unicorn" suggests it could be; please link to things like this; not everyone knows what "unicorn" is), you need to do some debugging to work out where the DNS issues are. Is it the recursive DNS server your app is using? Some closer caching server? Some intermediate server up the chain used to resolve Heroku's DNS (unlikely) ? Heroku's DNS its self? Is it a packet loss issue, an issue where the server responds with a false negative, one bad server in a round-robin cluster, ... ? Expect to spend lots of quality time with dig +trace and with traceroute or mtr - assuming you can ssh in. If you have root, tcpdump and tshark may also be useful, as will examination of the logs of the DNS server you're using if you can get access to them.

If you have no shell access you pretty much have to bug the support of the people you're hosting with.

Solution 2

Just for reference, I had the same problem (Heroku was having an issue with AWS). Here was the helpful response from customer service (08/15):

We are occasionally seeing DNS resolution failure on our instances and we're working to solve this with AWS. The issue is really intermittent and unfortunately we're having hard time trying to identify the cause. While we're investigating, I have one workaround for this issue - currently, you're using Django's default way for the database connection - which is trying to make a new connection per request. This is pretty expensive, also this means you'll need to do DNS resolution for your every request which would increase the possibility to face this issue very unfortunately. Instead, I recommend that you use connection pooling. You can find some examples here:

https://devcenter.heroku.com/articles/python-concurrency-and-database-connections

One customer who was seeing this issue occasionally (but frequently, with Django app) saw a good improvement with the connection pooling, this should help a lot to prevent it from happening.