Linux – Systemd restart service if it’s not listening to a port

debianlinuxmonitoringsystemd

I've been using Systemd for a while in Debian 8. I use the Restart=on-failure option to waken-up a service in case of failure.

I will like to know if there is a way to force the restart of the service if it is not listening to a specific port (even if the process stills running).

The reason of why I need that is because we are developing the new feature to will fix this issue but it will require a while. Meanwhile we need a workaround.

I have developed a script to check that status:

#! /bin/bash
PORTS=( 1452 542 )
for port in ${PORTS[@]}; do
    netstat -anp | grep $port > /dev/null 2>&1
    if [ "$?" -ne 0 ]; then
        # Port blocked. Kill the running process and start it again after a while
done

This script is triggered periodically using the cron.I know it is a dirty trick. That's why I would like to integrate that behaviour in Systemd checking. Is that possible?

Thank you in advance.

Cheers,

A.

Best Answer

I would suggest you approach it differently and use a dedicated monitoring tool to accomplish this.

My favourite monitoring tool which allows services to be restarted in case they crash, or are not listening at their configured port(s) anymore is monit: https://packages.debian.org/jessie/monit

There are plenty of examples in the configuration file, on this site and elsewhere about setting it up. I will list an example to give you an idea how simple it is to use, the example uses init.d however it can easily be converted to use systemd.

To test a process exists and is listening on a specific port and to start it when this test fails:

check process example with pidfile /var/run/example.pid
start program = "/etc/init.d/example start"
start program = "/etc/init.d/example stop"
if failed host 192.0.2.10 port 80 protocol http then restart
if 5 restarts within 5 cycles then timeout

You can leave out the protocol http part and monit will do just a simple tcp connect to test it out. The protocol argument does a more involved test to check whether something is actually responding to, for example, an http get request.

You need to make sure the process or service is started in such a way a corresponding pid file is created in /var/run. Monit itself doesn't take care of that. Normally if the service is started through an init script or systemd it should have a pid file in /var/run.

Related Topic