Centos – 404 in Tomcat manager with AJP connector

ajpcentoshttp-status-code-404mod-jktomcat

I'm trying to configure Tomcat 7 on CentOS 6 with Apache mod_jk, SSL, and an AJP connector.

The JK module is installed under Apache:

$ sudo httpd -M | grep jk
jk_module (shared)
Syntax OK

Tomcat is configured with an AJP connector as follows:

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" packetSize="65536" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

The workers.properties file is configured as follows:

worker.list=default,lb,jk-status
worker.default.type=ajp13
worker.default.host=x.x.x.x
worker.default.port=8009
worker.lb.type=lb
worker.lb.balance_workers=default
worker.jk-status.type=status

Apache is configured to serve Tomcat webapps through a virtual host as follows:

LoadModule jk_module modules/mod_jk.so

JkWorkersFile /etc/httpd/conf/workers.properties
JkShmFile /var/log/httpd/mod_jk.shm
JkLogFile /var/log/httpd/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +RejectUnsafeURI +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"
JkMountCopy All

<VirtualHost x.x.x.x:80>
  ServerAdmin webmaster@my.domain.com
  ServerName my.domain.com
  ServerAlias www.my.domain.com

  # Always redirect to SSL
  RewriteEngine on
  ReWriteCond %{SERVER_PORT} !^443$
  RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R,L]
</VirtualHost>

<VirtualHost x.x.x.x:443>
  ServerAdmin webmaster@my.domain.com
  ServerName my.domain.com:443
  ServerAlias www.my.domain.com

  ErrorLog /var/log/httpd/ssl_my.domain.com_error_log
  LogFormat "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %>s %b" ssl_log
  CustomLog /var/log/httpd/ssl_my.domain.com_access_log ssl_log

  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/my.domain.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/my.domain.com.key
  SSLCertificateChainFile /etc/pki/tls/certs/tw-chain.crt

  JkMountCopy On
  JkMount /manager default
  JkMount /manager/* default
</VirtualHost>

Tomcat and the manager app are up and running with no apparent errors:

Sep 11, 2014 12:29:18 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.31 using APR version 1.3.9.
Sep 11, 2014 12:29:18 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Sep 11, 2014 12:29:18 PM org.apache.catalina.core.AprLifecycleListener initializeSSL
INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
Sep 11, 2014 12:29:18 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-apr-8009"]
Sep 11, 2014 12:29:18 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 512 ms
Sep 11, 2014 12:29:18 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Sep 11, 2014 12:29:18 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.55
Sep 11, 2014 12:29:34 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/share/apache-tomcat-7.0.55/webapps/manager
Sep 11, 2014 12:29:34 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /usr/share/apache-tomcat-7.0.55/webapps/manager has finished in 146 ms
Sep 11, 2014 12:29:34 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-apr-8009"]
Sep 11, 2014 12:29:34 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 15586 ms

Yet when I try to load the manager app at https://x.x.x.x/manager, I get a 404 error. I know the request is getting through to Tomcat, because the URL redirects to /manager/html, and I get the Tomcat 404 instead of the Apache 404. Apache logs show the requests:

[17/Sep/2014:15:44:05 -0400] x.x.x.x TLSv1.2 AES256-SHA256 "GET /manager/html HTTP/1.1" 404 975
[17/Sep/2014:15:44:06 -0400] x.x.x.x TLSv1.2 AES256-SHA256 "GET /manager/html HTTP/1.1" 404 975

Logs for mod_jk show the requests too:

[Wed Sep 17 15:43:54 2014] [22722:140360872835040] [info] init_jk::mod_jk.c (3383): mod_jk/1.2.40 initialized
[Wed Sep 17 15:44:05 2014] default x.x.x.x 0.001380
[Wed Sep 17 15:44:06 2014] default x.x.x.x 0.001489

Curl provides a little more info:

$ curl -kLI https://x.x.x.x/manager
HTTP/1.1 302 Found
Date: Wed, 17 Sep 2014 18:50:22 GMT
Location: https://x.x.x.x/manager/
Content-Type: text/plain; charset=UTF-8

HTTP/1.1 302 Found
Date: Wed, 17 Sep 2014 18:50:22 GMT
Set-Cookie: JSESSIONID=D4B4A040BE7D4724A7D25B7F63FC2B92; Path=/manager/; Secure; HttpOnly
Location: https://x.x.x.x/manager/html;jsessionid=D4B4A040BE7D4724A7D25B7F63FC2B92
Content-Type: text/html;charset=ISO-8859-1

HTTP/1.1 404 Not Found
Date: Wed, 17 Sep 2014 18:50:22 GMT
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 1063

I'm stumped by this one – everything seems to be functioning properly, yet somewhere along the line the communication is breaking down. Any insights would be greatly appreciated, thanks!

Best Answer

The problem turned out to be a failed build of the APR library. I was using Ansible to deploy Tomcat, and an apparent bug in the "shell" command caused the APR configuration to fail silently.

Details of this issue with Ansible can be seen here: Ansible playbook not working trying to run make & configure with complex switches