Ubuntu – How to make upstart back off, rather than give up

Ubuntuupstart

I want Upstart to do two things:

  1. stop trying to respawn a failed process so fast
  2. never give up trying to respawn

In an ideal world, upstart would try to restart a dead process after 1s, then double that delay on each attempt, until it reached an hour.

Is something like this possible?

Best Answer

The Upstart Cookbook recommends a post-stop delay (http://upstart.ubuntu.com/cookbook/#delay-respawn-of-a-job). Use the respawn stanza without arguments and it will continue trying forever:

respawn
post-stop exec sleep 5

(I got this from this Ask Ubuntu question)

To add the exponential delay part, I'd try working with an environment variable in the post-stop script, I think something like:

env SLEEP_TIME=1
post-stop script
    sleep $SLEEP_TIME
    NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
    if [ $NEW_SLEEP_TIME -ge 60 ]; then
        NEW_SLEEP_TIME=60
    fi
    initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
end script

** EDIT **

To apply the delay only when respawning, avoiding the delay on a real stop, use the following, which checks whether the current goal is "stop" or not:

env SLEEP_TIME=1
post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [ $goal != "stop" ]; then
        sleep $SLEEP_TIME
        NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
        if [ $NEW_SLEEP_TIME -ge 60 ]; then
            NEW_SLEEP_TIME=60
        fi
        initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
    fi
end script
Related Topic