Java process restarting prematurely under runit

javalogstashrunit

I'm attempting to run Logstash under supervision with runit. My run script for Logstash is simply

#!/bin/sh
SSL_CERT_DIR=/etc/ssl/certs exec /usr/bin/java -jar /usr/local/bin/logstash.jar agent -f /tmp/logstash.conf --log /var/log/logstash.log

If I run this script manually it takes 30 or 45 seconds for the process to properly start up and start doing work, but it will stay up afterwards for a couple hours. However, when I try to do sv start logstash it executes the script (and Logstash definitely starts because I see output written to /var/log/logstash.log), but after 15 or 20 seconds runit seems to kill and restart the Logstash java process.

I've tried setting the -w option, but it doesn't seem to wait any longer for the process to run.

How can I have runit properly let this Java process process run and only restart it when it dies?

Best Answer

First, install svlog. Using it is fairly painless.

Second, (assuming the service definition lives at /etc/sv) change your /etc/sv/logstash/run to say:

#!/bin/sh
exec 2&>1
SSL_CERT_DIR=/etc/ssl/certs exec /usr/bin/java -jar /usr/local/bin/logstash.jar agent -f /tmp/logstash.conf --log /var/log/logstash.log

You are right in that stderr must be redirected to stdout.

Then, do this (adapting to your installation as needed):

mkdir /etc/sv/logstash/log
touch /etc/sv/logstash/log/run
chmod 744 /etc/sv/logstash/log/run
mkdir /var/log/logstash
chown log /var/log/logstash
chmod 775 /var/log/logstash
ln -s /var/log/logstash /etc/sv/logstash/log/main

This creates a log entry for the service, sets up a stub file to be edited, creates a directory to hold the log files, and provides a symlink for the logging service to divert output to.

Finally, put this in /etc/sv/logstash/log/run

#!/bin/sh
exec 2>&1
exec svlog -tt main

Restart your service with sv restart logstash and wait a few seconds...and you should have two processes listed, your logstash service and the svlog running with it. All of the output from logstash will be fed into svlog, which in turn will write a log file to the /etc/sv/logstash/log/main symlink, which in turn drops off in /var/log/logstash/current. Terminating the logstash service with sv stop logstash will allow any final output to flush into svlog which will write to disk before exiting.

Related Solutions

Centos – Logstash Forwarder doesn’t start up with chkconfig in CentOS 5

I ended up posting about the issue on the github page for the project, and I got quite a speedy response pointing me to another issue which came along with another init script, which I have now implemented. This seems to work correctly.

Thanks to TrevorH in the CentOS IRC channel for the assitance, as well as driskell on github for getting me to the new init script, I'll copy it below for reference in case anyone has the same problem I have with that script above:

#!/bin/bash
# chkconfig: 345 80 20
# description: Logstash Forwarder
# processname: logstash-forwarder
# config: /etc/logstash-forwarder
# pidfile: /var/run/logstash-forwarder.pid

### BEGIN INIT INFO
# Provides: logstash-forwarder
# Required-Start: $local_fs $network $remote_fs
# Required-Stop: $local_fs $network $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop logstash-forwarder
# Description: Logstash Forwarder
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

PATH=/sbin:/usr/sbin:/bin:/usr/bin

prog=logstash-forwarder
DAEMON=/opt/logstash-forwarder/bin/logstash-forwarder
pidfile=/var/run/$prog.pid
lockfile=/var/lock/subsys/$prog

# load defaults

[ -e /etc/default/$prog ] && . /etc/default/$prog
[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

DAEMON_ARGS="${DAEMON_ARGS:--config /etc/logstash-forwarder -spool-size 100 -log-to-syslog}"

start()
{
    echo -n $"Starting $prog: "
    nohup $DAEMON $DAEMON_ARGS </dev/null >/dev/null 2>&1 &
    retval=$?
    pid=$!
    echo $pid > $pidfile
    if [ rh_status_q ]; then
        touch $lockfile
        success
        echo
    fi
    return $retval
}

stop()
{
    echo -n $"Stopping $prog: "
    killproc -p "$pidfile" $prog
    retval=$?
    [ -f "$pidfile" ] && rm -f $pidfile
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
    status -p $pidfile $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    status)
        rh_status
        ;;
    *)
        echo "Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

This has been modified slightly as the original script had .conf in the DAEMON_ARGS statement which wasn't needed on CentOS 5 or 6

EDIT

There were some issues with the init script, but the commiter has updated it today, for my instance I also needed either of the following files:

/etc/defaults/logstash-forwarder
/etc/sysconfig/logstash-forwarder

Content:

DAEMON_ARGS="${DAEMON_ARGS:--config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100}"

Linux – Logstash: Failed to flush outgoing items

To re-import date, remove $HOME/.sincedb file, remove indexes in elastic search , and set start_position to beginning:

 input {
 #apache
 file {
     type => "apache-access"
     path => "/var/log/apache2/access.log"
     start_position => beginning
 }

Best Answer

Related Solutions

Centos – Logstash Forwarder doesn’t start up with chkconfig in CentOS 5

Linux – Logstash: Failed to flush outgoing items

Related Topic