Passenger + nginx: application doesn't run in production mode

Question

I'm using Passenger + Nginx to run a Rails app. If i use "rails_env development;" the application works fine.

But if i run in production mode, i get "We're sorry, but something went wrong.".

I did ran db:migrate for production and i can access the database normally.

The strange thing is that i don't get any new entries in logs ( nginx and rails one ) and i made sure nginx user could write on them.

If i run rails console production, it works fine:

# rails console production
Loading production environment (Rails 3.2.0)
1.9.3-p125 :001 > 

Any ideas of what might be happening? What else should i check?

----Edit----

After @BenLee suggestion of adding passenger_debug_log_file to nginx.conf i started to get this error for both development and production:

# /etc/init.d/nginx restart
nginx: the configuration file /opt/nginx/conf/nginx.conf syntax is ok
nginx: [alert] Unable to start the Phusion Passenger watchdog: it seems to have crashed during startup for an unknown reason, with exit code 1 (-1: Unknown error)
nginx: configuration file /opt/nginx/conf/nginx.conf test is successful
Parando o nginx:                                           [  OK  ]
Iniciando o nginx: nginx: [alert] Unable to start the Phusion Passenger watchdog: it seems to have crashed during startup for an unknown reason, with exit code 1 (-1: Unknown error)
                                                           [  OK  ]

I searched about this error but i still haven't found a solution. Watchdog seems to exist and is in the right place:

# ls -lh /usr/local/rvm/gems/ruby-1.9.3-p125/gems/passenger-3.0.11/agents/
total 7,8M
drwxr-sr-x 2 root rvm 4,0K Fev 22 23:34 nginx
-rwxr-xr-x 1 root rvm 4,6M Fev 22 23:35 PassengerLoggingAgent
-rwxr-xr-x 1 root rvm 3,3M Fev 22 23:34 PassengerWatchdog

If i remove passenger_debug_log_file it starts normally. But the original problems comes back, application does not run in production mode.

Solution 1

This looks like it could be some sort of file corruption, possibility located in one of the agents. One thing that may help is to reinstall fresh versions of the agents. You can do this via the passenger standalone package.

To do this, first SSH in to the server and then, in any directory (in the examples below, I'm going to assume your home directory, but any directory will do), run:

passenger package-runtime

This will seem to do a lot of downloading and configuration and installation, but don't be alarmed. All that will do is install the passenger standalone version in a subdirectory called passenger-standalone. After running this command, you should see a file structure like this (assuming you ran the command in your home directory):

+ /home/you
    + passenger-standalone
        + 3.0.11-....
            nginx-x.y.z.tar.gz
            support.tar.gz

So it's just a tree with two tar.gz files in it (note the 3.0.11-... is not literal, but will start with that string -- the full name is system dependent; similarly the x.y.z in one of the tar.gz files will actually be a version number). Next, untar the support.tar.gz, doing something like this:

cd passenger-standalone/3.0.11-....
tar xzvvf support.tar.gz

This will extract to the current directory. Among many other things, it will create an agents subdirectory with two new files:

+ /home/you
    + passenger-standalone
        + 3.0.11-....
            + agents
                PassengerLoggingAgent
                PassengerWatchdog

The idea is to copy these agent files over your original system ones, to fix any corruptions that may be present in the system version. But first, set the permissions appropriately. Following from where the commands above left off, run:

cd agents
sudo chown root:rvm Passenger*

Next, backup your original agents:

cd /usr/local/rvm/gems/ruby-1.9.3-p125/gems/passenger-3.0.11/agents
sudo cp PassengerLoggingAgent PassengerLoggingAgent.backup
sudo cp PassengerWatchdog PassengerWatchdog.backup

Finally, move the fresh versions from the standalone package here.

sudo mv /home/you/passenger-standalone/3.0.11-..../agents/Passenger* .

Then, restart nginx. If for some reason you need to undo this, just restore the backup copies and restart nginx again.

As an optional cleanup step, you don't need those standalone files any more, so you can remove them like this:

rm -rf /home/you/passenger-standalone

Solution 2

Even though Fernando seems to have solved the problem through updating the agents I had a similar issue but another solution.

I also had this behavior with a rails app when using passenger and nginx. Everything worked fine when setting rails_env development; in nginx.conf. However when changing development to production it did not work anymore and the connection was refused when trying to connect from the browser. Anyway, the problem was in the production.rb file and the line

config.force_ssl = true

When I commented that out, the application worked in production. And instead of configuring ssl in the rails app I did it in nginx.conf.

Hope this might help someone else setting up a rails app with passenger and nginx.

Comments

[nickgrim]

what does rails server -e production do?

[Fernando]

It starts normally, listening on port 3000.

[qweet]

If you find that the Rails logs aren't being written to, check the error log of nginx (/var/log/nginx/error.log). They should have something in there that should give you a clue as to why this is happening.

[Ben Lee]

I find that the important log file to check to debug this kind of problem is passenger.log, not error.log

[Fernando]

@BenLee where can i find passenger.log? Is it create inside nginx logs folder?

[Ben Lee]

@Fernando, look at your nginx.conf for a line that looks like passenger_debug_log_file. That will tell you where the log file is.

[Fernando]

Thanks @BenLee. I have updated the question with more information. I'm not sure, but i haven't seem anyone with this exact error code. Maybe it's time to fill in a bug report.

[Ben Lee]

@Fernando, I have some ideas, but I need more info. Can you post your full nginx.conf? (just erase any sensitive information)

[Ben Lee]

Also, please post the output of this command on the server machine: ls -la /tmp | wc -l (this will tell you how many files are in the /tmp directory)

[pjmorse]

Fernando, with your recent edit, you say, "application does run in production mode." Do you mean, "application does NOT run in production mode"? That was your original problem, right?

[pjmorse]

Also, can you post the nginx vhost configuration for your site? Not necessarily the whole nginx.conf file, but the server block for your production site.

[Fernando]

@BenLee Sorry for the delay. My nginx.conf is available here pastebin.com/BcN4EQD1

[Fernando]

@pjmorse That is correct. It was a typo in my last edit. I've fixed it. See my previous response to BenLee with my nginx conf.

[Fernando]

@BenLee My tmp folder contains 31 files at this moment.

[Ben Lee]

@Fernando, none of my ideas were right :(. I'm out of ideas about what's going wrong, it really looks like some file just got corrupted somehow. If I were in your shoes at this point, I'd try re-installing all the agents by copying them from a standalone installation. Do you know how to do that? If not, I'll post it as an answer (because it may solve your problem).

[gokva]

you should check for conflicting versions of ruby, and also that your log file folders have the right permissions. No logs is a sign that the daemon doesn't have permissions.

[Fernando]

@BenLee I'm not sure about how to do that. If you can add this as an answer i would appreciate.

[Fernando]

Thanks for all the help. I was able to fix the problem following your answer.