Ssh – causing the famous “broken pipe” when I use tmux inside an SSH session

linux-networkingsshtmux

Scenario1:
I SSH from my employer (OSX Yosemite SSH client) to my home server (Debian stable) and (re)connect to a tmux session. If I leave the terminal inactive for a while (~20minutes), it seems like the network connection has been interrupted for too long with the famous SSH error: "broken pipe". If I log back in the tmux behaves as expected: the session is still alive.

Scenario2: I SSH from my employer to my home box. Now I do the stuff I usually do but this time not in a tmux session. Now I can leave the terminal inactive for hours and hours, it does not die.

Scenario 3: I SSH from my own home network segment to the same server and connect to a tmux session. Now I get the expected behaviour. The terminal does not lose the connection after a period of inactivity, with our without using tmux.

I have tried to Google my question but the problem is that I get answers to the wrong question:
Q: I use SSH for a long running command, how can I reconnect if I get a broken pipe?
A: Have you heard of tmux?
Yes I have … . But I haven't heard of broken pipes being caused by tmux.

I also asked the question on irc but I didn't get a reply there.

Best Answer

Enable SSH keepalives globally in /etc/ssh_config or in your home directory ~/.ssh/config

KeepAlive yes
ServerAliveInterval 60

Before I found this openssh option, I would run "top -d 60" to keep the connection alive.

Related Solutions

Linux – Why does the ssh timeout vary with network location

There is a good writeup on this problem here.

They recommend:

ssh -o TCPKeepAlive=yes

or:

ssh -o TCPKeepAlive=no -o ServerAliveInterval=15

However, I do have an issue at my work site where I get disconnected from sessions, where at home they are fine. I believe my firewall (SonicWall) may be flucking with TCPKeepAlive, maybe because of NAT.

My SSH client, SecureCRT, fortunately has an option for a "NO-OP" protocol, which I believe basically sends a command that doesn't do anything to the server. By manually enabling this I am able to remain connected. Not sure what the MacOSX terminal client has that is similar to that. There is a writeup on how to implement "NO-OP" on command line.

Finally, you may wish to use Wireshark or other sniffer to watch your actual TCP connection to find out what is going on with it. That would be the final way to see why it is still disconnecting occasionally.

SSH – Adding Port Forwardings Programmatically on a ControlMaster SSH Session

That's quite simple, actually. Simply add the ctl_cmd -O forward to your existing command, thus:

ssh -M -L5555:localhost:22 remotehost

becomes:

ssh -O forward -M -L5555:localhost:22 remotehost

The ssh man page discusses the -O ctl_cmd option:

-O ctl_cmd
        Control an active connection multiplexing master process.  When the -O option is
        specified, the ctl_cmd argument is interpreted and passed to the master process.
        Valid commands are: “check” (check that the master process is running), “forward”
        (request forwardings without command execution), “exit” (request the master to
        exit), and “stop” (request the master to stop accepting further multiplexing
        requests).

This, of course, assumes you've either enabled ControlMaster yes in your ~/ssh/config file or -M on the command line.

Related Topic