Tomcat Error – Resolving 400 Error for Specific Hosts

It smells like maybe it's running into a mod_rewrite or alias before it's getting to your virtual host definition. Everything you've posted looks correct, but you can try adding a few extra options:

JkWorkersFile  /explicit/path/to/workers.properties
JkLogFile      /var/log/mod_jk.log
JkLogLevel     debug

That first setting will make sure you're using the actual properties file you think you are, maybe something in your multi-tomcat environment is loading the wrong one. You might also check this specific instance's TOMCAT_HOME to ensure it's loading the right server.xml; you can actually pass that as well as a parameter to $TOMCAT_HOME with -config:

cd $TOMCAT_HOME
./bin/startup.sh -config /direct/path/to/server.xml

See where that takes you...

There may be a few things going on here. Your command above translates to 50 concurrent connections, each issuing 1000 requests. One thing to note here is that if I recall correctly apachebench does not enable keep alive by default. It may be worth adding this (pass -k to your command above). This will be more of a real world test anyway, as most user agents do use keep-alive, as does Tomcat, by default. This should help the issue if my theories below are correct.

1) I suspect that your slamming that thread pool with too many requests, since each one is tearing down. This is a pretty big hit to those threads, as well as the TCP/IP stack on the system. Which leads me to...

2) You may be (ok, you probably are) running out of ephemeral ports and or hitting TIME_WAIT sockets. If each request is indeed a new, unique request, you're very likely going to be running into a TIME_WAIT situation with thousands of sockets in that state (have a look at netstat -an |grep -ic TIME_WAIT for a count of them during your load). These sockets will be ineligible for re-use unless you've enabled time_wait_reuse on your system. The fact that you're using localhost only makes this worse.

For more information on setting time_wait reuse up, have a look here. Also note that this thread correctly points out that setting the fin_wait timeout is incorrect in the context of time_wait, so avoid that. Tickling fin_wait in the context of TIME_WAIT is wrong and won't help you.

So have a look and potentially tweak tcp_tw_recycle/reuse specifically. These will help you get through your tests, as will keep-alive.