first of all: I am no Tomcat expert, so please bear with me.
Currently I'm facing an issue when I reboot the aforementioned server. The tomcat application is not available (404 not found) when automatically started.
As soon as I stop the Tomcat via systemd and start it manually via "/usr/share/tomcat9/bin/catalina.sh" the configured application (from server.xml) is available and accessible.
./catalina.sh start
Using CATALINA_BASE: /var/lib/tomcat9
Using CATALINA_HOME: /usr/share/tomcat9
Using CATALINA_TMPDIR: /var/lib/tomcat9/temp
Using JRE_HOME: /usr/lib/jvm/java-12-oracle
Using CLASSPATH: /usr/share/tomcat9/bin/bootstrap.jar:/usr/share/tomcat9/bin/tomcat-juli.jar
some excerpts from /etc/tomcat9/server.xml
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
Server=" " />
....
<Host name="host-live" appBase="webapps/host-live"
unpackWARs="true" autoDeploy="true">
<Alias>some.domain.com</Alias>
<Context path="" docBase="/var/lib/tomcat9/webapps/host-live" />
<Valve className="org.apache.catalina.valves.RemoteIpValve" remoteIpHeader="x-forwarded-for" />
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="some.domain.com_access" suffix=".txt"
pattern="%t %{x-forwarded-for}i %a %m %U %r %b %s %D %I %{X-AUSERNAME}o %{User-Agent}i" resolveHosts="false" />
</Host>
My goal is to have it started and accessible automagically at boot time 🙂
Hence I tried to modify systemd unit file. I changed it to the following:
# Current Configuration
Environment="CATALINA_HOME=/usr/share/tomcat9"
Environment="CATALINA_BASE=/var/lib/tomcat9"
Environment="CATALINA_TMPDIR=/tmp"
Environment="JAVA_OPTS=-Djava.awt.headless=true"
EnvironmentFile=-/etc/default/tomcat9
# My changes:
# old values:
#ExecStart=/bin/sh /usr/libexec/tomcat9/tomcat-start.sh
# replaced with
ExecStart=/usr/share/tomcat9/bin/startup.sh
ExecStop=/usr/share/tomcat9/bin//shutdown.sh
EDIT:
I fixed some permission problems for the tomcat logfiles. But still the application is not found 404 error.
Seems like Tomcat is deploying the application successfully when started by systemd but its still not available. Error 404 still coming up on request http://subdomain.domain.tld/myapp/MyAppServer
The funny thing is: the application is instantly available on a cloned server (local hostfile edited). I dont get it.
Mar 24 13:20:00 server rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Mar 24 13:20:03 server tomcat9[848]: Match [Server/Service/Engine/Host/Valve] failed to set property [resolveHosts] to [false]
Mar 24 13:20:03 server tomcat9[848]: Match [Server/Service/Engine/Host/Valve] failed to set property [resolveHosts] to [false]
Mar 24 13:20:03 server tomcat9[848]: Server version name: Apache Tomcat/9.0.16 (Ubuntu)
Mar 24 13:20:03 server tomcat9[848]: Server built: Sep 11 2019 19:47:51 UTC
Mar 24 13:20:03 server tomcat9[848]: Server version number: 9.0.16.0
Mar 24 13:20:03 server tomcat9[848]: OS Name: Linux
Mar 24 13:20:03 server tomcat9[848]: OS Version: 4.15.0-1063-aws
Mar 24 13:20:03 server tomcat9[848]: Architecture: amd64
Mar 24 13:20:03 server tomcat9[848]: Java Home: /usr/lib/jvm/java-12-oracle
Mar 24 13:20:03 server tomcat9[848]: JVM Version: 12.0.1+12
Mar 24 13:20:03 server tomcat9[848]: JVM Vendor: Oracle Corporation
Mar 24 13:20:03 server tomcat9[848]: CATALINA_BASE: /var/lib/tomcat9
Mar 24 13:20:03 server tomcat9[848]: CATALINA_HOME: /usr/share/tomcat9
Mar 24 13:20:03 server tomcat9[848]: Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
Mar 24 13:20:03 server tomcat9[848]: Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
Mar 24 13:20:03 server tomcat9[848]: Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djava.awt.headless=true
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -XX:+UseG1GC
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Xms1024M
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Xmx6144M
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Dignore.endorsed.dirs=
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Dcatalina.base=/var/lib/tomcat9
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Dcatalina.home=/usr/share/tomcat9
Mar 24 13:20:03 server tomcat9[848]: Command line argument: -Djava.io.tmpdir=/tmp
Mar 24 13:20:03 server tomcat9[848]: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
Mar 24 13:20:03 server tomcat9[848]: Initializing ProtocolHandler ["http-nio-8080"]
Mar 24 13:20:03 server tomcat9[848]: Server initialization in [1,490] milliseconds
Mar 24 13:20:03 server tomcat9[848]: Starting service [Catalina]
Mar 24 13:20:03 server tomcat9[848]: Starting Servlet engine: [Apache Tomcat/9.0.16 (Ubuntu)]
Mar 24 13:20:05 server tomcat9[848]: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 24 13:20:37 server tomcat9[848]: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [31,362] milliseconds.
Mar 24 13:20:37 server tomcat9[848]: Deploying web application directory [/var/lib/tomcat9/webapps/my-app-test/my-app]
Mar 24 13:20:38 server tomcat9[848]: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 24 13:20:38 server tomcat9[848]: Deployment of web application directory [/var/lib/tomcat9/webapps/my-app-test/my-app] has finished in [1,029] ms
Mar 24 13:20:39 server tomcat9[848]: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 24 13:20:39 server tomcat9[848]: Deploying web application directory [/var/lib/tomcat9/webapps/my-app-live/my-app]
Mar 24 13:20:40 server tomcat9[848]: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 24 13:20:40 server tomcat9[848]: Deployment of web application directory [/var/lib/tomcat9/webapps/my-app-live/my-app] has finished in [1,106] ms
Mar 24 13:20:40 server tomcat9[848]: Starting ProtocolHandler ["http-nio-8080"]
Mar 24 13:20:40 server tomcat9[848]: Server startup in [36,550] milliseconds
Mar 24 13:22:04 server tomcat9[848]: Pausing ProtocolHandler ["http-nio-8080"]
Mar 24 13:22:04 server tomcat9[848]: Stopping service [Catalina]
Mar 24 13:22:04 server tomcat9[848]: Stopping ProtocolHandler ["http-nio-8080"]
Mar 24 13:22:04 server tomcat9[848]: Destroying ProtocolHandler ["http-nio-8080"]
Best regards,
Marc
Best Answer
Since my application seems to work as expected when launched from a new server from an image, I checked the port tomcat was listening at (tcp 8080) as suggested. I could see tomcat throwing error 404 on port 8080, and was first able to reproduce the issue with a test server.
I then checked the permissions of the deployed applications inside the webapps directory and found owner group being root:root
Hence I fixed permissions with the following commands:
This made the webapplication working as expected when launched by systemd.
Thanks a ton for your suggestions. Stay healthy.