Starting Tomcat 8.5 using systemd on Centos 7

Question

I installed Tomcat 8.5 and I am able to start it successfully manually, for example:

su tomcat startup.sh

works properly and tomcat is able to serve on port 8080.

I need to automate the startup, so I created the file /etc/systemd/system/tomcat-prod.service:

[Unit]
Description=Tomcat 8.5 servlet container - Production
After=network.target

[Service]
Type=forking

ExecStart=/opt/appservers/production/apache-tomcat-8.5.37/bin/startup.sh
ExecStop=/opt/appservers/production/apache-tomcat-8.5.37/bin/shutdown.sh

User=tomcat
Group=tomcat   

[Install]
WantedBy=multi-user.target

The file is bare on purpose, all my env variables are in setenv.sh. I tried running it using:

# systemctl daemon-reload
# systemctl start tomcat-prod

For some reason above results in Tomcat exiting just after startup. I get no log anywhere that explains why. Tomcat creates an empty catalina.out file on startup and systemctl status tomcat-prod gives only the following:

tomcat-prod.service - Tomcat 8.5 servlet container - Production Loaded: loaded (/etc/systemd/system/tomcat-prod.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2019-01-04 08:08:27 UTC; 3s ago Process: 3583 ExecStop=/opt/appservers/production/apache-tomcat-8.5.37/shutdown.sh (code=exited, status=203/EXEC) Process: 3569 ExecStart=/opt/appservers/production/apache-tomcat-8.5.37/bin/startup.sh (code=exited, status=0/SUCCESS) Main PID: 3581 (code=exited, status=0/SUCCESS)

Jan 04 08:08:27 *.net startup.sh[3569]: Existing PID file found during start. Jan 04 08:08:27 *.net startup.sh[3569]: Removing/clearing stale PID file.

Can someone point me in the right direction?

Solution 1

Systemd keeps track of the "main process" of every service, so that it could accurately know whether the service is still alive or not. When the main process exits, the service manager assumes that the service has crashed or just stopped on its own. And when the service has stopped, all its leftover processes are automatically killed, to provide a clean state for future starts.

For Type=forking services, determining which is the "main" process can be somewhat difficult, especially when it's hidden behind layers and layers and layers of shell scripts. The default heuristics in systemd work well for regular daemons, but in your case they likely misdetect one of those shellscript layers as the main process, instead of tomcat itself. So as soon as the script ends, the service is assumed to have stopped and its leftovers (including the real daemon) are killed.

You have two options:

• Explicitly specify the main process using a "pidfile". This is the traditional method used by SysVinit. Your startup scripts already store the Tomcat daemon's process ID in some file, so your tomcat.service unit needs to reference it using the PIDFile= option. With this, the service manager will not need autodetection heuristics and will just track the provided PID as the main process.

• Get rid of the shellscripts entirely and start Tomcat directly from the ExecStart= option. This is the recommended method for systemd, and you can find example .service units already written by various distributions' packagers. For example, Arch Linux (Tomcat 8 with Jsvc).

Solution 2

No doubt grawity is already aware of the systemd horror mentioned elsewhere in which the author writes "we already have a better dæmon supervisor. It's systemd" - which would mean we don't need jsvc either. On that page, however, the author does add a "Bonus Track": "Notice that M. Benjamin invokes catalina.sh with run rather than with start. There is actually another path through catalina.sh, for systems that aim to run dæmons, not to spawn them." Adding a few tweaks to Jonathan de Boyne Pollard's script led me to the following which a) is not too hard to write, b) appears to eliminate unnecessary layers in functionality, the unnecessary use of PIDs and of additional daemonizing commands and c) is clear in its output:

# file: /usr/local/lib/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat 9
After=network.target

[Service]
User=tomcat
Group=tomcat

ExecStart=/opt/apache/tomcat/bin/catalina.sh run
Type=simple

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

For additional options refer to the top of the catalina.sh script where it reads:

#   Do not set the variables in this script. Instead put them into a script
#   setenv.sh in CATALINA_BASE/bin to keep your customizations separate.

Comments

[john]

I love down votes without a comment to explain why ...

[Aulis Ronkainen]

Can you please try it with the envs added to the .service file? Does the same error occur? The shutdown.sh file is also in bin directory, if you haven't moved it.

[john]

@AulisRonkainen the /bin was a typo above, corrected. It was correct in the actual script. I tried moving the env into the script but the result is the same. No log and it does not start.

[john]

All this makes sense, thx. First time i encountered systemd, upgrading from old distribution with chkconfig.

[john]

Well, that solved it.

[Dave Teezo]

Useful, thank you, that works fine! Could you explain why you are disabling SIGKILL and disabling the stop timeout? Are those two decisions related?

[zielot]

@Dave Teezo - using sigterm allows tomcat to shut down gracefully: "I also confirmed with the tomcat community that using SIGTERM to gracefully shutdown tomcat is fine (it's not functionally different than the Bootstrap.stop() call) so I think that's the way we should go, if you agree." see bugzilla.redhat.com/show_bug.cgi?id=1347864