Tomcat – Apache2 mod_jk LoadBalancer + Tomcat7 – Performance-Tuning

apache-2.2load balancingmod-jktomcat

we have a very high load System with round about 8.5 Million User per day. Actually we have 8 Apache2 Loadbalancer (Ubuntu 10.04.3 LTS x64 via DNS-RoundRobin) which are sending the traffic via AJP1.3 to 20 Workers. On this workers our self-written Grails-Webapplication is running on a tomcat7 + java7 server (Ubuntu 11.10 x64). The Application is Logging to a local postgresql 9.1 database. All Servers are hostet on Hetzner Germany an have an Intel® Core™ i7-2600 with 16GB's RAM an 2x3TB Raid 1 HDD.

At Prime-Time (6(pm) till 12(pm)) we have sometimes serious performance bottlenecks but the hardware is not the Problem. CPU is max at 50%, load max. 2.5 and RAM only used 3-4GB max. But i have no Idea, where is the Problem.

Here is an excerpt of my server.xml:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector  port="8009"
            protocol="AJP/1.3"
            maxThreads="5000"
            minSpareThreads="25"
            maxSpareThreads="75"
            acceptCount="200"
            redirectPort="8443"
            connectionTimeout="60000"
            debug="4"
/>

At peak the Tomcats handle 400-450 Accesses per second. Have anyone some tips how to optimize the performance of Apache + mod_jk + tomcat7?

Best Answer

Garbage collection would be my guess -- it can cause bottlenecks on loaded Java servers. Have you studied your garbage collection logs to get an idea of the delays involved? What are your garbage collection settings

Related Solutions

Tomcat – Apache, Tomcat and mod_jk for load balancing

"balancer://" is used by apache mod_proxy_balancer and probably with mod_proxy_ajp who is another way to connect apache to tomcat ajp than mod_jk.

I can't tell you a lot on the load detection reliability but I learned this from my experience :

Read carrefully the mod_jk directives manual
Check the timeout parameters, this is very important. Default infinite "reply_timeout" is very bad: if a tomcat is dead apache will keep threads busy and you can be flooded.
Read the mod_jk logs to look for errors
Let apache handle directly static content with JkUnMount directive to give tomcat only needed requests.
Use apache mod_cache if possible
Use the mod_jk watch page to check what is happening on your connectors.

An article on mod_jk/proxy-ajp: http://thoughts.contegix.com/2009/02/02/mod-proxy-ajp-versus-mod-jk-round-1/

Tomcat – Apache mod_jk Setting for Tomcat – workers.properties

You are pretty close on your config, there are just a few things that are off.

First, AJP does not run over port 8080, by default it runs over port 8009. It's a different connector. You can verify which port AJP is setup for by looking at your server.xml file for a line that looks like this:

<Connector port="8009" redirectPort="8443" protocol="AJP/1.3"/>

So you'll want to change your worker.ajp13.port=8080 directive to worker.ajp13.port=8009

Secondly, your httpd.conf is a bit off:

JkAutoAlias <path_to_tomcat>/webapps
JkMount /<your_webapp>/*.jsp ajp13

The Tomcat connector docs explain JKAutoAlias very well:

Automatically Alias webapp context directories into the Apache document space. Care should be taken to ensure that only static content is served via httpd as a result of using this directive. Any static content served by httpd will bypass any security constraints defined in the application's web.xml. For inheritance rules, see: JkMountCopy. There is no default value.

Related Topic