We've got a number of web services running through Tomcat which use hibernate/mysql. I suspect that some of them have incorrectly configured connection pools, as after several hours some of the individual apps run out of connections and stop responding. We have been making changes to the connection pool service (in this case, C3P0), but we still need to leave the old versions of the applications on the server for backwards compatibility.
At any rate, I suspect that these applications are also hurting tomcat's overall stability. About once a week, our server stops responding entirely and is unable to serve even static pages. After restarting the service, everything works again for another few days or so. Looking through the logs reveals hardly any uncaught exceptions, so I'm not sure what would be causing tomcat to crash. Nothing remarkable is listed in the error logs before the server quits responding, sadly.
We are also considering switching to JBoss, as it is a bit more "enterprisey", but I'm not convinced it will solve these problems. Is there any compelling reason to switch web platforms, or should I debug further within our own webapps? Also, is it possible for the webapp to crash the application server by doing something bad?
Server config: Windows 2003 Server, Tomcat 6.0.18 + blazeDS 3.0, Hibernate 3.2.
Best Answer
I don't think anybody will have the answer to your problem, but only leads & ideas. Here are some:
you need robots that will check the health of each part of your service. (testing a single connection to your database, getting a static web page, getting a dynamic web page...). This way, you'll see what breaks first or response time increase.
do you have a monitoring/stats service? You need to keep track of the "number of active database connections", "number of active web sessions", "number of tomcat threads", "memory available", CPU...
My advice, there is no tomcat process left because they're all stuck waiting for a resource (maybe a database connection, or they're just an infinite loop!). The tools I listed before will definitely help you understand why your server is dying slowly every week.
netstaton your server and look at the number of connections to the database server (and check it against your pool size and database server capacity).