Linux – Phantom process killing

centoscentos5javakilllinux

I have a java application running on a CentOS 5.4 server.

Randomly, and seemingly without cause, these processes will die, not through the application exiting, or due to my killing it, but due to something that seems to kill without leaving a trace.

I log all output from the application, as well as sending stderr and stdout to a logfile, and none of these output logs contain anything that would indicate why these processes have died.

My first instinct was the kernel-level OOM killer, but the system is never low on memory, and usually has somewhere between 1GB and 3GB of memory free at any given point in time.

My question is: does anyone know what might be causing it, or does anyone have any ideas where I could start looking?

Thanks.

Best Answer

Starting my java app with 

strace -tt -o strace.out <java command>

Showed that it was being sent a SIGHUP command which was killing it. Turns out that, although I was setting the process to run in the background (by appending & to the command), the command is still tied to whichever thread created it.

This meant that, for commands executed by apache, they would be killed whenever the worker thread that created them was recycled, and, for commands executed manually, they were killed whenever I logged out.

I remedied this by simply pre-pending nohup to the command, i.e.

nohup java -jar /path/to/my/java.jar arguments &
Related Topic