systemctl stops Tomcat service immediately after start

Question

I asked first this question on StackOverflow, but it seems that this site is a better fit.

I am setting up a web app environment to run Java apps. The environment is supposed to be:

• AWS EC2 Ubuntu 18
• Java 8
• Tomcat 9

I launched an EC2 instance and installed java, then I followed this guide in order to set up Tomcat: https://www.digitalocean.com/community/tutorials/install-tomcat-9-ubuntu-1804

The problem now rises when systemctl tries to start/restart Tomcat service. Here is what I see when checking out systemctl logs:

Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: tomcat.service: Service hold-off time over, scheduling restart.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: tomcat.service: Scheduled restart job, restart counter is at 5.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Stopped Apache Tomcat Web Application Container.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Starting Apache Tomcat Web Application Container...
Dec 19 13:35:25 ip-10-165-10-74 startup.sh[1705]: Tomcat started.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Started Apache Tomcat Web Application Container.
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: Dec 19, 2018 1:35:26 PM org.apache.catalina.startup.Catalina stopServer
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: Dec 19, 2018 1:35:26 PM org.apache.catalina.startup.Catalina stopServer
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: SEVERE: Error stopping Catalina
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: java.net.ConnectException: Connection refused (Connection refused)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.PlainSocketImpl.socketConnect(Native Method)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.connect(Socket.java:589)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.connect(Socket.java:538)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.(Socket.java:434)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.(Socket.java:211)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.lang.reflect.Method.invoke(Method.java:498)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:403)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:497)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: The stop command failed. Attempting to signal the process to stop through OS signal.
Dec 19 13:35:27 ip-10-165-10-74 shutdown.sh[1725]: Tomcat stopped.

The systemd config:

[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking

Environment=JAVA_HOME=/usr/lib/java/jdk1.8.0_191
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms256M -Xmx512M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

As I understand the problem here is that Tomcat cannot connect to 8005 port in order to listen to shutdown commands.

However, I opened this port in EC2 Security Group console: 8005 127.0.0.1/32

The other strange thing is that Tomcat starts okay when rebooting ec2 instance, but fails to restart throwing the exception like above.

What could be a problem here? How do I check that port 8005 is actually open?

===========

Alright, I made some progress, I figured out that Tomcat seems to work fine. startup.sh works okay - you may see this in the log - it echoes "Tomcat started". This is the last line in catalina.sh start. However, I don't understand why systemctl tries to stop the service immediately after it starts it. It sends catalina.sh stop command.

Solution 1

Thanks @MichaelHampton for the good article: https://jdebp.eu/FGA/systemd-house-of-horror/tomcat.html

It explains the possible problems with a typical installation suggested in the tutorial I read (and tons of other copy-pasted ones). I could not figured out why the problem arises. Maybe it is the problem with all of the layers introduced by catalina.sh, startup.sh, etc.

So basically, I rewrote the systemd config script and it started to work.

Solution 2

I had the exact same problem. Only I was using Amazon Lightsail. Maybe there is a problem with the way that Amazon configures their Ubuntu machines, and the described way to create a Tomcat Service. I used the described way on a local Ubuntu machine that I set up myself and that worked fine.

I read the article by @michael-hampton and used the approach described there. I am a Linux noob, and I found that I had some problems getting it to work. I'm including the files I ended up with here so maybe other Linux noobs that have this problem can get an easier start. My Tomcat installation is at /opt/tomcat and these are the files I ended up with:

/etc/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat Web Application Container
After=network.target


[Service]
User=tomcat
Group=tomcat
UMask=0007

EnvironmentFile=/etc/default/tomcat
ExecStart=/usr/bin/env ${JAVA_HOME}/bin/java \
$JAVA_OPTS $CATALINA_OPTS \
-classpath ${CLASSPATH} \
-Dcatalina.base=${CATALINA_BASE} \
-Dcatalina.home=${CATALINA_HOME} \
-Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS} \
-Djava.io.tmpdir=${CATALINA_TMPDIR} \
-Djava.util.logging.config.file=${CATALINA_BASE}/conf/logging.properties \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
org.apache.catalina.startup.Bootstrap \
start

ExecStop=/usr/bin/env ${JAVA_HOME}/bin/java \
$JAVA_OPTS $CATALINA_OPTS \
-classpath ${CLASSPATH} \
-Dcatalina.base=${CATALINA_BASE} \
-Dcatalina.home=${CATALINA_HOME} \
-Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS} \
-Djava.io.tmpdir=${CATALINA_TMPDIR} \
-Djava.util.logging.config.file=${CATALINA_BASE}/conf/logging.properties \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
org.apache.catalina.startup.Bootstrap \
stop

SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

/etc/default/tomcat

CATALINA_HOME=/opt/tomcat
CATALINA_BASE=/opt/tomcat
CATALINA_TMPDIR=/opt/tomcat/temp
CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC
JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom
JAVA_ENDORSED_DIRS=/opt/tomcat/endorsed
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
CLASSPATH=/opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar

Comments

[happybuddha]

Thank you. This helped me. I signed up to this stack only to upvote. Cheers.