Ssh – Reliable Reverse SSH Tunnel

sshssh-tunnel

I'm using autossh to open a reverse SSH tunnel using private keys. The tunnel user's shell is rssh and is verified as working.

The problem I'm having is as such, regardless of ClientAliveInterval (15) and ClientAliveCountMax (3) (the sshd defaults), on the client side the tunnel is opened with:

exec autossh -N                   \
  -o "ServerAliveInterval 60"     \
  -o "ServerAliveCountMax 3"      \
  -o "StrictHostKeyChecking no"   \
  -R ${tunnel_port}:localhost:22  \
  -i ./tunnel                     \
  -v                              \
  tunnel@directory.mytunnelhost.com

The ${tunnel_port} comes from a small curl call which looks for a free port on the tunnel host, the port doesn't change (it's assigned based on the mac address of the requesting box)

The problem is that when TERMinating the tunnel client, the ports stay occupied on the host, and the sshd process keeps running:

enter image description here

I also can't log into the machines via the tunnel:

ubuntu@ip-10-252-138-233:~$ ssh -p 39777 pi@localhost
ssh: connect to host localhost port 39777: Connection refused

I'm wondering what I can do to make the tunnel more reliable, to make sure when either end goes down that it's completely dead, torn down and can't enter a "broken" state.

If I was tunnelling over port 80 or similar, I could write a check script (this daemon os started with runit) to check that some HTML page was available end-to-end through he tunnel, however when it's over SSH, and as a reverse tunnel, I'm at a loss as to how the client could know if the tunnel is really working as against just running?

Best Answer

I think you have two questions here.

First is how to make sure that the server sshd processes terminate, freeing up the ports, once the client disconnects. If the client disconnects then I don't know why the server processes would hang around, unless maybe they have child processes, but your screenshot shows that they don't. If the client doesn't disconnect but just stops responding, then the server process should terminate by about ClientAliveInterval*ClientAliveCountMax seconds later.

Second is how the client can determine whether the tunnel is working. This is what autossh's port monitoring feature does. If you add e.g. -M 20000 to the call to autossh, it will set up port forwards to send traffic to the remote host on port 20000, and receive it back on port 20001. It will try to send traffic around the loop every AUTOSSH_POLL seconds (default 600), and if it detects that the connection has stopped working, terminate it and start a new ssh connection.

Adding the -M switch to your autossh call, and probably setting AUTOSSH_POLL lower, to say 30, may help if ssh hasn't been terminating correctly.

Related Topic