Linux – Hung Java JVM failing to respond to kill -3

javalinuxthreads

I have a Java VM that is hanging "randomly". I quote the randomly bit, because there is obviously a reason that the VM is hanging, but the hang does not occur periodically. We have the same software running in different customer environments and in those environments the JVM is not hanging.

In the process of attempting to troubleshoot the hang the process exists with zero CPU utilization. I then attempt to execute kill -3 and the kill command hangs. No JVM Thread Dump is produced.

I have spent time instrumenting the code to periodically log the thread stack traces hoping to catch the JVM in a state that would indicate where the issue lies, but so far this attempt has not born much fruit.

Unfortunately I have not been able to reproduce this issue in my lab environment so I am limited by what can be done at the Customer site.

The OS's in question are Red Hat Enterprise 5.4 and SUSE 10 running java version 1.6.0_05-b13

Has anyone had this problem? Any ideas on why kill -3 is failing to produce a Java Thread Dump?

Thanks!

Best Answer

Sounds like you might be running out of heap space.

Even if your JVM does not produce thread dumps, you should be able to generate a heap dump. This, with the GC stats, will also give you insight into what is going. Consult your JVM's documentation for how to generate heap dump in binary format. Then download Memory Analyzer and open the heap dump file to see what is taking up memory. Typically, one would use the jmap command with the proper options. Generating the heap dump will cause your JVM to slow down quite a bit; I've seen it take 30 mins to generate a heap dump 1.5GB large, YMMV.

Also, post your java opts, both from the servers where it is hanging and working fine.

Related Topic