SSH connection keeps crashing

firewallssh

although this is not a real programming question perhaps anyone has any idea regarding my problem.

It's pretty weird. When I remotely connect one of our ubuntu servers per ssh the connection can be established without any problems.

I log in and enter something that produces output on the console like "ls -al". Then the connection just hangs. If I enter "ls" it shows me the output. If I enter "dmesg" session crashes too.

There's nothing in the servers log and the terminal on my side doesn't recognize the connection being closed it just hangs.

Now I wonder if this could be Firewall-Related. If there's to much output sent back it crashes.

Or maybe one of our programmers messed up something. I really don't know in which direction I can try to track this problem. Any ideas on this? I just can't understand what is happening and if it's related to network problems or system misconfiguration….I can't see why there's a difference between "ls" and "ls -al" or "dmesg". And the login produces a lot of output too on the screen but it always works.

Ok I just found out about serverfault.com and posted my question there. This can be deleted because it's off topic here. Thx anyway!

Best Answer

What you've likely stumbled upon is a PMTU problem.

Many devices have a Maximum Transmission Unit(MTU) which is the maximum size of the packet they can send. If the device sees a packet beyond this threshold it will break it up into smaller parts, but some packets have a special flag that says Don't Fragment(DF). These packets are dropped and a ICMP message is sent to the sender so that the send can resize the packet itself.

In your case it looks like there is something between the SSH server and you that is dropping these packets and not telling the server. If you have a packet dump of the connection you will probably see packets missing. There is a tool on Linux called tracepath that lets you see the mtu of all the hops on the way to the server.

To work around this issue you should add an iptables rule such as the following to the network your server resides in:

iptables -A INPUT -d x.x.x.x/24 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --clamp-mss-to-pmtu

Related Solutions

Ssh – Why does ssh hang after “debug1: loaded 3 keys”

This is a symptom I have seen at regular interval, and I'd wager it is caused by one of the hops between your cloud server and the user's network filtering out ICMP messages indiscriminately, causing path MTU discovery to fail; the key exchange is the first part of the SSH protocol where it is likely that large TCP segments are sent over the socket, and most likely to trigger the problem.

Two easy ways to isolate the problem: try to make large exchanges over some other protocol between the same hosts (for instance, upload some file over a web form), or artificially force the MTU on the user's host to some value small enough to fit any plausible MTU along the network (~300b). If the problem pops up during other large transfers and disappears with the smaller MTU, then you've found the culprit.

(The reason that happens is that when an IP packet that has the DF (don't fragment) flag set is transmitted that does not fit through a hop, the hop will respond with an ICMP message; and filtering those ICMP messages out is a common misconfiguration by inexperienced or overly paranoid network admins - so the sending host never knows the path MTU is smaller than it thinks and packets just vanish in the aether).

The bad news is that this is not something that can be fixed at either endpoint in general, but at the hop where the improper filtering is taking place. You can work around the problem with manually configured MTUs, but that's an ugly hack and is brittle at best.

Ssh – How to automate SSH login with password

Don't use a password. Generate a passphrase-less SSH key and push it to your VM.

If you already have an SSH key, you can skip this step… Just hit Enter for the key and both passphrases:

$ ssh-keygen -t rsa -b 2048
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.

Copy your keys to the target server:

$ ssh-copy-id id@server
id@server's password:

Now try logging into the machine, with ssh 'id@server', and check-in:

.ssh/authorized_keys

Note: If you don't have .ssh dir and authorized_keys file, you need to create it first

to make sure we haven’t added extra keys that you weren’t expecting.

Finally, check to log in…

$ ssh id@server

id@server:~$

You may also want to look into using ssh-agent if you want to try keeping your keys protected with a passphrase.

Best Answer

Related Solutions

Ssh – Why does ssh hang after “debug1: loaded 3 keys”

Ssh – How to automate SSH login with password

Related Topic