The website is under heavy load + ROR

Question

We are running a website with ROR on CentOS 6 with 2 web server and 1 database server. Some times it shows message "The website is under heavy load"... Can some plese help you what to check here.

We are using Passenger 4.0.21 with Ruby 1.8.7 and Apache 2.2.15. Web server is running with the default settings.

Below is some output of passenger-status:

# passenger-status

Version : 4.0.21
Date    : Thu Dec 12 02:02:44 -0500 2013
Instance: 20126
----------- General information -----------
Max pool size : 6
Processes     : 6
Requests in top-level queue : 0

----------- Application groups -----------
/home/web/html#default:
  App root: /home/web/html
  Requests in queue: 100
  * PID: 20290   Sessions: 1       Processed: 53      Uptime: 24h 3m 5s
    CPU: 0%      Memory  : 634M    Last used: 23h 16m 8
  * PID: 22657   Sessions: 1       Processed: 37      Uptime: 23h 15m 55s
    CPU: 0%      Memory  : 609M    Last used: 22h 44m
  * PID: 29147   Sessions: 1       Processed: 146     Uptime: 20h 47m 48s
    CPU: 0%      Memory  : 976M    Last used: 18h 20m
  * PID: 22216   Sessions: 1       Processed: 26      Uptime: 10h 3m 19s
    CPU: 0%      Memory  : 538M    Last used: 9h 44m 4
  * PID: 23306   Sessions: 1       Processed: 75      Uptime: 9h 43m 22s
    CPU: 0%      Memory  : 483M    Last used: 8h 44m 4
  * PID: 25626   Sessions: 1       Processed: 115     Uptime: 8h 46m 42s
    CPU: 0%      Memory  : 540M    Last used: 7h 59m 5

Solution

You have too many requests in queue. Since version 4.0.15 there is a limit which is 100 by default. Here is a short excerpt from http://blog.phusion.nl/2013/09/06/phusion-passenger-4-0-16-released/ which says:

Phusion Passenger now displays an error message to clients if too many requests are queued up, instead of letting them wait. This much improves quality of service. By default, "too many" is 100. You may customize this with PassengerMaxRequestQueueSize (Apache) or passenger_max_request_queue_size (Nginx).

Have a look at the user guide about this: http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMaxRequestQueueSize

You could try increasing it or setting it to 0in order to disable it.

EDIT

You should also check your logs to see whether there are requests which take too long. Maybe you have some processes in your code that take too long. I prefer using NewRelic for monitoring those things.