I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM:
http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm
When I went through the process I found the init script provided for CentOS/RHEL/Fedora didn't work as expected. I got no information fed back to me when running any commands and it didn't start the process. I found someone else had some similar problems and had written their own, this script works fine when I start up logstash-forwarder using service or the init.d command, however, I have noticed that when I reboot the server the service doesn't start up (the following is from just after a reboot):
# service logstash-forwarder status
logstash-forwarder dead but pid file exists
# chkconfig --list logstash-forwarder
logstash-forwarder 0:off 1:off 2:off 3:on 4:on 5:on 6:off
If I then start it manually, it all works fine. Some research has told me that I should have a file here (as I use run level 3):
# ls -la /etc/rc3.d/ | grep logstash
lrwxrwxrwx 1 root root 28 Nov 5 12:33 S99logstash-forwarder -> ../init.d/logstash-forwarder
Which is in place and is only followed by smartd in the numerical list, so I don't think the issue is that services that logstash-forwarder relies on are not fired up already. Can anyone advise how further to investigate in to why this isn't starting up?
EDIT
I have noticed this issue is not apparent on a CentOS 6 system, only on a CentOS 5 machine. I am not sure what difference this makes as the init script template files are essentially the same. Anyone?
FURTHER EDIT
I have had to modify the init script slightly, this was just down to the fact I was shipping /var/log/messages to my logstash server, but the init script made the logstash logs go in to messages, causing a feedback loop, every time it sent a log to logstash it would add a log to messages, which would then be sent to logstash. So I have changed the init script slightly:
#! /bin/sh
#
# chkconfig: 345 99 99
# description: logstash-forwarder
# processname: logstash-forwarder
# config: /etc/logstash-forwarder
# Source function library.
. /etc/init.d/functions
NAME=logstash-forwarder
DESC="log shipper"
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
DAEMON=/opt/logstash-forwarder/bin/logstash-forwarder
DAEMON_ARGS="-config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100"
start() {
echo -n "Starting $NAME: "
daemon --pidfile $PIDFILE "nohup $DAEMON $DAEMON_ARGS >/dev/null 2>&1 &"
touch /var/lock/subsys/$NAME
echo $(pidofproc $NAME) > $PIDFILE
echo
return 0
}
stop() {
echo -n "Shutting down $NAME: "
killproc -p $PIDFILE
rm -f /var/lock/subsys/$NAME
echo
return 0
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $PIDFILE $NAME
;;
restart)
stop
start
;;
*)
echo "Usage: service $NAME {start|stop|status|restart}"
exit 1
;;
esac
exit $?
This script works perfectly well on CentOS 6, but when I reboot in CentOS 5 I have the issue explained above, any help would be greatly appreciated.
ANOTHER EDIT
I have done some further testing, and found that when I reboot the machine logstash-forwarder does start up, but the logstash-forwarder.pid file is empty, so I guess the init script is saying it's dead just because it cannot find the process as it's not noted in the pid file. The following is from straight after a reboot:
# service logstash-forwarder status
logstash-forwarder dead but pid file exists
# ps -ef | grep logstash
root 3297 1 2 10:20 ? 00:00:02 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root 3722 3324 0 10:22 pts/0 00:00:00 grep logstash
# cat /var/run/logstash-forwarder.pid
# service logstash-forwarder restart
Shutting down logstash-forwarder: [FAILED]
Starting logstash-forwarder: [ OK ]
# cat /var/run/logstash-forwarder.pid
3748 3297
# ps -ef | grep logstash
root 3297 1 2 10:20 ? 00:00:03 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root 3748 1 1 10:22 pts/0 00:00:00 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root 3767 3324 0 10:22 pts/0 00:00:00 grep logstash
# service logstash-forwarder restart
Shutting down logstash-forwarder: [ OK ]
Starting logstash-forwarder: [ OK ]
# ps -ef | grep logstash
root 3788 1 5 10:22 pts/0 00:00:00 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/logstash-forwarder.conf -spool-size 100 -log-to-syslog
root 3796 3324 0 10:22 pts/0 00:00:00 grep logstash
# cat /var/run/logstash-forwarder.pid
3788
So it looks like the service starts, no entry is made in the pid file, restart the process I get 2 copies of the service running, as it doesn't kill the first process, but the second time I restart it both processes are killed and I am back to having one process.
Now I need to figure out why the pid file entry isn't altered when I boot the machine.
Best Answer
I ended up posting about the issue on the github page for the project, and I got quite a speedy response pointing me to another issue which came along with another init script, which I have now implemented. This seems to work correctly.
Thanks to TrevorH in the CentOS IRC channel for the assitance, as well as driskell on github for getting me to the new init script, I'll copy it below for reference in case anyone has the same problem I have with that script above:
This has been modified slightly as the original script had .conf in the DAEMON_ARGS statement which wasn't needed on CentOS 5 or 6
EDIT
There were some issues with the init script, but the commiter has updated it today, for my instance I also needed either of the following files:
Content: