Linux – How to kill a process that never dies

centos6linuxlogstash

Problem

I have java process which does not die neither with SIGTERM nor SIGKILL.

logstash  2591     1 99 13:22 ?        00:01:46 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

It respawns everytime signal is received.

Sep 15 13:22:17 test init: logstash main process (2546) killed by KILL signal
Sep 15 13:22:17 test init: logstash main process ended, respawning

It sounds strange but even I reboot the server, it still does not die.

Process was executed through init script with below command:

NAME=logstash
LS_USER=logstash
LS_OPTS="--path.settings=/etc/logstash"
LS_PIDFILE=/var/run/$NAME/$NAME.pid
LS_STDERR="/var/log/logstash/logstash.stderr"
DAEMON="/usr/share/logstash/bin/logstash"

runuser -s /bin/sh -c "exec $DAEMON ${LS_OPTS}" ${LS_USER} &>${LS_STDERR} &

Is there any way to force this process to kill other than reinstalling the OS?

Environment

Process : 

logstash 5.0.0~alpha5

OS :

Red Hat Enterprise Linux Server release 6.7 (Santiago)

Java version : 

openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

Server is deployed on Microsoft Azure.

Best Answer

init: logstash main process (2546) killed by KILL signal

Actually your process does stop here.

init: logstash main process ended, respawning

A new logstash process is started by init to replace it.

That is also shows which control process is responsible for restarting logstash: init. (On RHEL 6 and CentOS that is Upstart) Your process most likely gets started from either/etc/inittab or a drop-in file in /etc/init/logstash.conf (or similar) and should be controlled with the apropiate tool, initctl and not with kill.

Try initctl list to see if logstash is there.

Then initctl stop logstash will stop it.

Editing or removing the conf file in /etc/init will allow you to disable it persistently.

You might even be able to control the job with the service and chkconfig commands.

Related Topic