Main process exited, code=exited, status=7/NOTRUNNING

Question

I have a node application, running via a systemd service on Ubuntu 16.04

However, after some time the service is getting restarted automatically.

Here's what I get in the logs:

Mar 17 14:35:10 testmachine systemd[1]: myService.service: Main process exited, code=exited, status=7/NOTRUNNING
Mar 17 14:35:10 testmachine systemd[1]: myService.service: Unit entered failed state.
Mar 17 14:35:10 testmachine systemd[1]: myService.service: Failed with result 'exit-code'.
Mar 17 14:35:40 testmachine systemd[1]: myService.service: Service hold-off time over, scheduling restart.

From what I understand, the systemd detects that the service is not running and restarts is after 30 seconds, as I have specified in the service config file.

myService.service

[Unit]
Description=The will start node app
After=cleanMongo.service

[Service]
ExecStart=/media/path/to/execution/scriptFile
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

But my issue is, why does it think the service is not running? The application is doing its thing and in the middle it restarts.

PS: I have lots of CRON jobs running in the node application, is it the case that the process is too busy to reply to some kind of isAlive from the systemd? And thats why systemd thinks its not running?

Solution

Generally, systemd/systemctl would report the failure to start (if/when you run sudo systemctl start myService.service).

I would check:

• the output of sudo systemctl status myService.service
• the output of sudo journalctl -ln 2000 -u myService

The output of either may help diagnose the issue (journalctl may be the better bet).

Also worth trying is replicating what your systemd service unit is doing - i.e. trying to run the same command from an interactive session (potentially using sudo) and seeing what errors are reported.

systemd/systemctl actually tracks the PID, so polling should not be an issue.

Updated based on your service definition, running the wrapper script could be one part of the issue.

Your systemd config expects to be able to track the process itself by PID. However, it would 'see' the PID of the script, but, depending on how you run node from that script, it may not be able to track node's PID.

It seems a more solid approach would be to run node directly (as suggested here):

[Unit]
Description=Node.js Example Server
#Requires=After=mysql.service       # Requires the mysql service to run first

[Service]
ExecStart=/usr/local/bin/node /opt/nodeserver/server.js
#WorkingDirectory=/opt/nodeserver   # Required on some systems
Restart=always
RestartSec=10                       # Restart service after 10 seconds if node service crashes
StandardOutput=syslog               # Output to syslog
StandardError=syslog                # Output to syslog
SyslogIdentifier=nodejs-example
#User=<alternate user>
#Group=<alternate group>
Environment=NODE_ENV=production PORT=1337

[Install]
WantedBy=multi-user.target

This way, systemd is able to track the correct PID.

If you need to run some commands before running node, consider using systemd's ExecStartPre.

There are a number of other options systemd provides (see the unit file man page), that may enable you to avoid using a wrapper:

• as mentionned, ExecStartPre allows you to perform tasks before the main process is started (and ExecStartPost to perform tasks after)
• you can express dependencies to other services via Require, Wants, Before, and After, and you can also use Conflicts or various Condition* keywords.

Comments

[Dushyant Bangal]

The messages I posted ARE output of journalctl. Before that it just had my application logs, executing cron jobs, etc

[Dushyant Bangal]

I wanted to move to the approach you posted, but there were a couple of other tasks in my script file. Like disabling another service first, then starting my application

[Dushyant Bangal]

The WorkingDirectory is not working for me. I have used path.resolve() at some places for config files which are in a the root of my project directory. But its not able to find the file.