Ssh – Why does ssh hang after “debug1: loaded 3 keys”

ssh

Trying to log in to an Amazon EC2 instance running Ubuntu 10.04.1. I can log in just fine, no issues. A different user, coming from a different network just gets this:

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to xxxx [xxxx] port 80.
debug1: Connection established.
debug1: identity file /.ssh/identity type -1
debug1: identity file /.ssh/id_rsa type -1
debug1: identity file /.ssh/id_dsa type -1
debug1: loaded 3 keys

And then it hangs.

We've tried running sshd on port 22 and port 80

I'm guessing that it's not a firewall problem since the verbose output reports that the connection is established.

I don't see anything in /var/log/auth.log when the failing user connects. I do see entries when I log in successfully.

Best Answer

This is a symptom I have seen at regular interval, and I'd wager it is caused by one of the hops between your cloud server and the user's network filtering out ICMP messages indiscriminately, causing path MTU discovery to fail; the key exchange is the first part of the SSH protocol where it is likely that large TCP segments are sent over the socket, and most likely to trigger the problem.

Two easy ways to isolate the problem: try to make large exchanges over some other protocol between the same hosts (for instance, upload some file over a web form), or artificially force the MTU on the user's host to some value small enough to fit any plausible MTU along the network (~300b). If the problem pops up during other large transfers and disappears with the smaller MTU, then you've found the culprit.

(The reason that happens is that when an IP packet that has the DF (don't fragment) flag set is transmitted that does not fit through a hop, the hop will respond with an ICMP message; and filtering those ICMP messages out is a common misconfiguration by inexperienced or overly paranoid network admins - so the sending host never knows the path MTU is smaller than it thinks and packets just vanish in the aether).

The bad news is that this is not something that can be fixed at either endpoint in general, but at the hop where the improper filtering is taking place. You can work around the problem with manually configured MTUs, but that's an ugly hack and is brittle at best.

Related Topic