Linux – systemd service shuts down on its own

bashlinuxredhatsystemd

I have a problem with this SystemD service:

[Unit]
Description=RTC Client Services
After=rds.service
Requires=rds.service

[Service]
User=USER
Group=GROUP
PermissionsStartOnly=true
RuntimeDirectory=rtc_client
RuntimeDirectoryMode=0770
WorkingDirectory=/usr/lib/systemd/scripts/
Type=forking
ExecStartPre=/bin/mkdir -p /var/run/rtc_client
ExecStartPre=/bin/chown -R USER:GROUP /var/run/rtc_client
ExecStart=/bin/bash rtc_client.sh start
ExecStop=/bin/bash rtc_client.sh stop
Restart=no
PIDFile=/var/run/rtc_client/rtc_client.pid
TimeoutStartSec=0
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

The machine boots every morning. The service does the ExecStart but suddenly it stops, as it tries to kill the process PID:

rtc_client.service - RTC Client Services
   Loaded: loaded (/usr/lib/systemd/system/rtc_client.service; enabled)
   Active: failed (Result: exit-code) since Thu 2016-06-23 06:25:46 CEST; 3h 33min ago
  Process: 2754 **ExecStop**=/bin/bash rtc_client.sh stop (code=exited, status=0/SUCCESS)
  Process: 1819 **ExecStart**=/bin/bash rtc_client.sh start (code=exited, status=0/SUCCESS)
  Process: 1815 ExecStartPre=/bin/chown -R USER:USER /var/run/rtc_client (code=exited, status=0/SUCCESS)
  Process: 1813 ExecStartPre=/bin/mkdir -p /var/run/rtc_client (code=exited, status=0/SUCCESS)
 Main PID: 1949 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/rtc_client.service

Jun 23 06:25:46 zprds60 bash[2754]: Database Connection Information
Jun 23 06:25:46 zprds60 bash[2754]: Database server        = DB2/LINUXZ64 10.5.5
Jun 23 06:25:46 zprds60 bash[2754]: SQL authorization ID   = USER
Jun 23 06:25:46 zprds60 bash[2754]: Local database alias   = DBALIAS
Jun 23 06:25:46 zprds60 bash[2754]: /home/pers5i/.bash_profile: line 97: unalias: vi: not found
Jun 23 06:25:46 zprds60 bash[2754]: USER IS:  root
Jun 23 06:25:46 zprds60 bash[2754]: PID IS:  1949
**Jun 23 06:25:46 zprds60 bash[2754]: rtc_client.sh: line 34: kill: (1949) - No such process**
Jun 23 06:25:46 zprds60 bash[2754]: logout
Jun 23 06:25:46 zprds60 systemd[1]: Unit rtc_client.service entered failed state.

Here's the script that rtc_client.service launches:

#!/bin/bash

RTCENGINEID=$HOSTNAME'_engine'
RTCUSER='RTCUSER'
RTCPW='RTCPWD'
RTCSERVER='server.example.com'
RTCSERVERPORT='????'
RTCREPOSITORY=https://$RTCSERVER:$RTCSERVERPORT/ccm
WORKDIR='/opt/ibm/buildsystemtoolkit/buildsystem/buildengine/eclipse'
JAVACMD=/opt/ibm/java-s390x-71/jre/bin/java
ARGS="-cp ./plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar org.eclipse.equinox.launcher.Main -application com.ibm.team.build.engine.jazzBuildEngine -repository $RTCREPOSITORY -engineId $RTCENGINEID -userId $RTCUSER -pass $RTCPW"
RTCJAR=org.eclipse.equinox.launcher
PIDFILE='/var/run/rtc_client/rtc_client.pid'
DEBUGLOG='/tmp/rtc_debug.log'

. /home/USER/.bash_profile

start() {

        cd $WORKDIR
        nohup $JAVACMD $ARGS > $DEBUGLOG &
        sleep 5
        pgrep -f $RTCJAR > $PIDFILE
        echo "USER IS: " $(whoami) | tee -a $DEBUGLOG
        echo "PID IS: " $(cat $PIDFILE) | tee -a $DEBUGLOG

}


stop() {

        echo "USER IS: " $(whoami) | tee -a $DEBUGLOG
        echo "PID IS: " $(cat $PIDFILE) | tee -a $DEBUGLOG
        kill $(cat $PIDFILE)
        rm -f $PIDFILE

}

restart() {

        stop
        start

}

reload() {

        restart

}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  reload)
        reload
        ;;
  *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
esac

exit $?

The weird thing is that if I start the service or reboot the machine during the day, the service starts and stays alive…

Loaded: loaded (/usr/lib/systemd/system/rtc_client.service; enabled)
   Active: active (running) since Thu 2016-06-23 10:21:44 CEST; 4s ago
  Process: 2754 ExecStop=/bin/bash rtc_client.sh stop (code=exited, status=0/SUCCESS)
  Process: 38200 ExecStart=/bin/bash rtc_client.sh start (code=exited, status=0/SUCCESS)
  Process: 38195 ExecStartPre=/bin/chown -R USER:GROUP /var/run/rtc_client (code=exited, status=0/SUCCESS)
  Process: 38194 ExecStartPre=/bin/mkdir -p /var/run/rtc_client (code=exited, status=0/SUCCESS)

Any help is much appreciated!

Best Answer

The reason your service is stopping is in the logging you provided:

/bin/bash rtc_client.sh start (code=exited, status=0/SUCCESS)

Your rtc_client.sh script is exiting with a SUCCESS status. You should investigate your script to see why it doesn't keep running continually.