Ssh – How to use nohup properly within an ssh session invoked by gitlab-runner

bashgitlabnohupssh

I use gitlab runner's ssh executor to access an embedded device via ssh. I want to run a script there.

Basically the script invokes an upgrade routine on the device using dbus. Simple enough, isn't it?

#!/bin/bash
dbus-send --session --print-reply --dest=blabla /path blabla.swupdate.InitiateUpdate string:"" string:"" string:"$1"
sleep 4
dbus-send --session --print-reply --dest=--dest=blabla /path blabla.UpgradeMode

Now the device will immediately reboot, causing the ssh connection with gitlab runner to terminate unexpectedly, which leads to a pipeline failure.

I came up which various script entries to detach the last dbus call from the ssh connection but to no avail.

Here is one thing I tried:

#!/bin/bash
dbus-send --session --print-reply --dest=blabla /path blabla.swupdate.InitiateUpdate string:"" string:"" string:"$1"
nohup $(sleep 2 && dbus-send --session --print-reply --dest=--dest=blabla /path blabla.UpgradeMode)&
echo "ready to disconnect ssh"

When I connect manually using ssh, executing the script and disconnect quickly manually using exit, I see the system reboot. But when gitlab-runner executes everything, the job succeeds, but the reboot does not happen. So I am sure that the nohup-command does not get fully executed.

I tried putting nohupin front of both the sleep and the dbus-send command, as well as trying to put disown after the parentheses. I also tried as suggested by a comment here, to put nohup in front of the script call in .gitlab-ci.yml.

None of that lead to success.

Any ideas, how to get this right?

Thanks

Best Answer

I'm sorry this is so late. I've recently just had the same problem. Gitlab runner blocks on child processes, and any process in the child tree. This makes the disown command not work. Forking, and nohup also don't work.

The only solution I could figure out was using the at command https://linux.die.net/man/1/at

Basically I put my command in a script then did:

at new < my_script.sh. That successfully exited and kicked off my program in the background.