Ssh – Remote synchronous-then-asynchronous execution with SSH

gnu-screennohupssh

TL;DR

I want to remotely execute a script that begins synchronously (so that the local ssh command fails if the preparatory commands fail) and then goes asynchronous, thus terminating the local ssh command.

Details

I have a script on a remote server that slowly downloads a multi-gigabyte file. I would like to launch this script via SSH from the local server with ssh remote-server 'nohup /path/to/script arguments' but kill the SSH connection when I know that the script has been successfully started the download. Once it is launched, the SSH connection doesn't serve any useful purpose, systematically fails somewhere during the downloads, and blocks the execution on the local server.

I can't just do ssh -f or ssh & because I need the command to fail on the local server server if the remote script doesn't launch, fails on the first commands before the download, or if the remote server is not reachable.

I have tried various nohup and screen tricks. The closest I got was the following:

  • Launched by the local server:

    ssh -t me@remote-server 'screen -S long-download /path/to/download.sh'

    or

    ssh -t me@remote-server 'nohup screen -S long-download /path/to/download.sh'

  • In download.sh launched on the remote server:

    preliminary-instructions
# if anything fails so far then the local server's ssh command should fail synchronously
download-command &
some-more-checking
screen -d long-download # we can now safely end the ssh session

But somehow screen still gets killed…

Sample reproduction code

  • On a remote server, in /path/to/test.sh:

    #!/bin/bash
# uncomment this line to validate synchronous failure
sleep 10 && echo foo >/tmp/bar &
screen -d long-download

  • Locally:

    ssh -t me@remote-server 'nohup screen -S long-download /path/to/test.sh'

Best Answer

Look at ssh subsystems. I do something similar, but not with a script (i.e. I use a.out/elf compiled programs.) But I just tested with a bash script and it works.

From the ssh man page:

     -s      May be used to request invocation of a subsystem on the remote
         system.  Subsystems are a feature of the SSH2 protocol which
         facilitate the use of SSH as a secure transport for other appli-
         cations (eg. sftp(1)).  The subsystem is specified as the remote
         command.

Use -s (lower case) on the client to specify the subsystem

Add another Subsystem entry to sshd_config on the server which points to your script. 

    # override default of no subsystems
Subsystem       logger  /usr/local/libexec/my-logger.sh
Subsystem       sftp    /usr/libexec/sftp-server

Your script should be started just as any other file with execute permissions. My example:

#! /usr/local/bin/bash
logger "testing 1.2.3"
echo "return text from my-logger"
exec 0>&- # close stdin
exec 0<&- 
exec 1>&- # close stdout
exec 1<&- 
exec 2>&- # close stderr 
exec 2<&- 
logger "subsystem: nohup new script"
nohup /path/to/another/script/logger2.sh &  
logger "subsystem: the subsystem started script is now done"
exit 0

You should be able to return useful error text for success or failure. The ssh connection (and thus stdin, stdout and stderr) is up until the subsystem ends it. The subsystem can continue to run or not.

Have your other script (logger2.sh above) sleep for 5-10 minutes so you can use netstat to see the ssh connection gone, ssh client back to the shell and e.g. ps ax show that the script is still running in the background. This script may also need to close fd's etc. It's just a quick example. Good luck.

Related Topic