Linux – systemctl stops Tomcat service immediately after start

amazon-web-serviceslinuxsystemctltomcatUbuntu

I asked first this question on StackOverflow, but it seems that this site is a better fit.

I am setting up a web app environment to run Java apps.
The environment is supposed to be:

  • AWS EC2 Ubuntu 18
  • Java 8
  • Tomcat 9

I launched an EC2 instance and installed java, then I followed this guide in order to set up Tomcat:
https://www.digitalocean.com/community/tutorials/install-tomcat-9-ubuntu-1804

The problem now rises when systemctl tries to start/restart Tomcat service. Here is what I see when checking out systemctl logs:

Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: tomcat.service: Service hold-off time over, scheduling restart.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: tomcat.service: Scheduled restart job, restart counter is at 5.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Stopped Apache Tomcat Web Application Container.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Starting Apache Tomcat Web Application Container...
Dec 19 13:35:25 ip-10-165-10-74 startup.sh[1705]: Tomcat started.
Dec 19 13:35:25 ip-10-165-10-74 systemd[1]: Started Apache Tomcat Web Application Container.
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: Dec 19, 2018 1:35:26 PM org.apache.catalina.startup.Catalina stopServer
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: SEVERE: Could not contact [localhost:8005] (base port [8005] and offset [0]). Tomcat may not be running.
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: Dec 19, 2018 1:35:26 PM org.apache.catalina.startup.Catalina stopServer
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: SEVERE: Error stopping Catalina
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: java.net.ConnectException: Connection refused (Connection refused)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.PlainSocketImpl.socketConnect(Native Method)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.connect(Socket.java:589)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.connect(Socket.java:538)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.(Socket.java:434)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.net.Socket.(Socket.java:211)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:513)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at java.lang.reflect.Method.invoke(Method.java:498)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:403)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]:         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:497)
Dec 19 13:35:26 ip-10-165-10-74 shutdown.sh[1725]: The stop command failed. Attempting to signal the process to stop through OS signal.
Dec 19 13:35:27 ip-10-165-10-74 shutdown.sh[1725]: Tomcat stopped.

The systemd config:

[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking

Environment=JAVA_HOME=/usr/lib/java/jdk1.8.0_191
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms256M -Xmx512M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

As I understand the problem here is that Tomcat cannot connect to 8005 port in order to listen to shutdown commands.

However, I opened this port in EC2 Security Group console:
8005 127.0.0.1/32

The other strange thing is that Tomcat starts okay when rebooting ec2 instance, but fails to restart throwing the exception like above.

What could be a problem here? How do I check that port 8005 is actually open?

===========

Alright, I made some progress, I figured out that Tomcat seems to work fine. startup.sh works okay – you may see this in the log – it echoes "Tomcat started". This is the last line in catalina.sh start. However, I don't understand why systemctl tries to stop the service immediately after it starts it. It sends catalina.sh stop command.

Best Answer

Thanks @MichaelHampton for the good article: https://jdebp.eu/FGA/systemd-house-of-horror/tomcat.html

It explains the possible problems with a typical installation suggested in the tutorial I read (and tons of other copy-pasted ones). I could not figured out why the problem arises. Maybe it is the problem with all of the layers introduced by catalina.sh, startup.sh, etc.

So basically, I rewrote the systemd config script and it started to work.