Upstart env stanza not setting environment variables (like NODE_ENV) for Node.js application

Question

I have an Upstart script for my server that looks like this:

description "myapp node.js server"

start on runlevel [2345]
stop on shutdown

env NODE_ENV=production
env CUSTOM=somevalue
exec sudo -u nodejs /usr/local/bin/node /opt/myapp/app.js >> /var/log/nodejs/myapp.log 2>&1

post-start script
    NODE_PID=`status myapp | egrep -oi '([0-9]+)$' | head -n1` 
    echo $NODE_PID > /var/run/nodejs/myapp.pid
end script

However, the app doesn't see NODE_ENV set to production. In fact, if I console.log(process.env) within the app, I don't see NODE_ENV or CUSTOM. Any ideas what's happening?

By the way, NODE_ENV=production node app.js works just fine.

Solution 1

From the sudo man page (Ubuntu version of sudo)

There are two distinct ways to deal with environment variables. By default, the env_reset sudoers option is enabled. This causes commands to be executed with a minimal environment containing TERM, PATH, HOME, SHELL, LOGNAME, USER and USERNAME in addition to variables from the invoking process permitted by the env_check and env_keep sudoers options. There is effectively a whitelist for environment variables.

Sudo is resetting the environment. This is a frustrating aspect of using su and sudo in upstart or init scripts. Recent versions of upstart support specifying uid/gid without the use of sudo via the setuid/setgid directives as in the example below. Also note the use of chdir.

start on filesystem and started networking
respawn
chdir /var/www/yourapp
setuid yourapp
setgid yourapp
env NODE_ENV=production
env PATH=/usr/local/bin:/usr/bin:/bin
env CUSTOM=somevalue
exec /usr/local/bin/node app.js | /usr/bin/multilog s1024000 /var/log/yourapp 2>&1

For older versions of upstart, here's what I used to do to work around it.

description "start and stop the example.com node.js server"

start on filesystem and started networking
respawn

chdir /path/to/your/code
exec su -c 'PATH=$PWD/node/bin NODE_ENV=$(cat node_env.txt) ./node/bin/node app/server.js' www-data  >> tmp/stdout.log 2>&1

Note that I just put a node_env.txt file in my app root that sets production mode, because I hate environment variables. You can just do NODE_ENV=production right there if you prefer.

Solution 2

Just for the record. The Upstart Cookbook recommends the usage of start-stop-daemon instead of su or sudo when your Upstart version does not implement setuid.

But, unless you are still using 10.04 LTS (Lucid Lynx) which only has Upstart version 0.6.5, you should be using the setuid/setgid directives.

Solution 3

This has been working for me to set node env variables in upstart.

#!upstart

start on runlevel [2345]
stop on runlevel [016]

respawn

script
    echo $$ > /var/run/app.pid
    exec sudo NODE_ENV=production /opt/node/bin/node /opt/myapp/app.js >> /var/log/app.sys.log 2>&1
end script

Comments

[Chris F]

Answer by Peter Lyons goes into depth and offers some best practice suggestions, but simply switching from "sudo -u" to "su -c" without touching anything else seems to solve this problem if I switch to using "www-data" user vs. my own "nodejs" user. The latter doesn't yet work, but that's somewhat irrelevant to this question.

[Chris F]

I should also mention that sudo -E (preserve environment) can be tried, though I haven't tested it myself.

[Chris F]

A few questions: (1) What's in node_env.txt, just "production"? (2) Why are you setting PATH to $PWD/node/bin? and using ./node/bin/node. Do you have a local install of node for your application? (3) Why use su -c vs. sudo -u? Thanks.

[Peter Lyons]

node_env.txt contains just the single word "production" (with a trailing newline). That's all. Yes I install node within my application root, which I consider a best practice. I have many applications on the same server and each needs a different node version. I never share versions for my app server stack. You should use "su" instead of sudo in upstart scripts because they are already run as root by the OS and you don't need sudo. "su" is what is appropriate here - become a different user. "sudo" is more about letting regular users do specific commands as root, which is not applicable here

[Chris F]

I switched the code to use exec su -c '...' nodejs >> ... and I now can't start the process. If I run that command manually in the shell, I am asked for a password. I tried the manual command with user www-data and it asked for a password. I then sudo su -c ... with www-data and it worked. I haven't yet tried using www-data in my Upstart script. What am I missing here about Linux user configurations?

[Peter Lyons]

That's how su works if you execute it as non-root. Run it as root and it won't prompt for a password. So to test interactively from the shell, first do "sudo su -" to become root, then run the su command as it is in your upstart file.

[Chris F]

I wasn't able to get it working with the "nodejs" user, but I did have success with the "www-data" user. I'll have to investigate more into what's going on with user permissions and passwords for my "nodejs" user.

[Ville]

For me this started working only after I removed the trailing newline from the node_env.txt.

[hostmaster]

This is not quite true (any more?) As of Upstart 1.4, Upstart has the ability to run a System Job as a specified user using the setuid and setgid stanzas.

[C2H5OH]

I was aware of that, thanks. But you made me realise that the answer was probably not clear enough.

[ppetraki]

Works for running celeryd as non-root user too.

[DGM]

I disagree with "environment variables are harmful" .. they provide a solid interface between code development and server deployment configuration. When coupled with a proper devops strategy such as chef, the environment variables can be reliably set in all the right places - making it easy for the deployment to alter the config without having to poke around inside the source code folders.