The difference between nf_conntrack_max and nf_conntrack_expect_max

I found a very, very, very helpful upstart script for people that are having problems in the future. Put this into /etc/init/

# /etc/init/debug.conf
start on ( starting JOB!=debug \
or started JOB!=debug \
or stopping JOB!=debug \
or stopped JOB!=debug )
script
exec 1>>/tmp/log.file
echo -n "$UPSTART_JOB/$UPSTART_INSTANCE ($0):$$:`date`:"
echo "Job $JOB/$INSTANCE $UPSTART_EVENTS. Environment was:"
env
echo
end script

This script basically logs all jobs that start or stop. I have found that CentOS 6 does not 'emit' anything about runlevels. (nor some of the other common events I had tried.'). Looking the log file that debug job creates in /tmp/log.file was very helpful. By changing the start of my script from:

start on runlevel [345]

to

start on started sshd

all my jobs appear to start up correctly. This was a pain in the rear, since every example I found used the former syntax..

Problem

The problem with the standard Tomcat shutdown script is that it is simply not tough enough. When you are using your distro's service scripts to stop Tomcat you'll eventually simply be calling Tomcat's own shutdown script. CentOS is no different in this respect. For this reason you need to be familiar with what Tomcat's shutdown script can do for you and most importantly what it doesn't do: It doesn't actually guarantee that Tomcat dies. Far from it.

The thing is that it is quite easy for Tomcat to get itself into some kind of situation where it cannot be killed via its management port .. or even via a TERM signal.

Let's review how Tomcat can be killed in order of escalation:

1. Killing Tomcat via its management port. This is the default and this is what Tomcat's own shutdown script attempts first. It won't work if the Java process is hung for one reason or another. All of the reports you see on the Internet where people complain that Tomcat won't stop is more or less always in this alley. This isn't really Tomcat's fault. Its quite easy to create a web-application that will make the whole container hang or impossible to stop via this method.

2. Killing Tomcat by sending the Java process a TERM signal. This is a stronger way to kill Tomcat than in (1) and Tomcat's own shutdown script will attempt this but its only activated if you've set the CATALINA_PID variable in your Tomcat setenv.sh. (which is highly recommendable in any case). Killing a Unix/Linux process by sending it a TERM signal is the default for the kill command from the OS. It is the polite way to tell a Unix/Linux process to die. Unfortunately even this will on rare occasions not kill the Tomcat process.

3. Killing Tomcat by sending the Java process a KILL signal. (from OS command line this would be kill -9 <pid>). This will always kill the process and should be the last resort. The problem here is that the standard Tomcat shutdown script will never attempt this even if method (1) and (2) fails. So if you really want to make sure that Tomcat has been killed then you have no choice but to implement your own wrapper script around Tomcat's own shutdown script.

If running Tomcat in a production environment you really need to consider if you can live with a situation where Tomcat doesn't die (or isn't restarted) when you run e.g. service tomcat restart. You may be doing this via cron and it that case you'll for sure expect a deterministic outcome, right?

Recommendations

• Always use a Tomcat setenv.sh file where you define CATALINA_PID. This will at least give you method #2 above. Tomcat's setenv.sh file doesn't exist by default so you must create it yourself.

• Create a wrapper script around Tomcat's own scripts that makes sure that Tomcat really dies.

At the place where I work we've implemented this on all hosts that run Tomcat as a service. Its the same problem/solution for any variant of Unix/Linux.