SSH/SFTP connections fails silently on OSX 10.8.2 – ssh-agent issue

mac-osxsftpsshssh-agent

I'm trying to login to a remote machine with SSH or SFTP.

  • when I try ssh u-indgo@ssh1.eu1.frbit.com the CLI just won't respond. I get an empty new line, in which I can type characters, but nothing more.
  • when I try to connect with SFTP using the same credentials (I use Transmit as my SFTP client) it just hangs forever and doesn't connect.

No errors. No response.

The problem isn't specific to frbit.com and persists with any other server I try to connect to.

ssh client debugging

Running the ssh client with the -vv flag I got the following output:

debug1: Reading configuration data /Users/matanya/.ssh/config
debug1: Reading configuration data /usr/local/Cellar/openssh/6.1p1/etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ssh1.eu1.frbit.com [46.137.57.195] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /Users/matanya/.ssh/id_rsa type 1
debug1: identity file /Users/matanya/.ssh/id_rsa-cert type -1
debug1: identity file /Users/matanya/.ssh/id_dsa type 2
debug1: identity file /Users/matanya/.ssh/id_dsa-cert type -1
debug1: identity file /Users/matanya/.ssh/id_ecdsa type -1
debug1: identity file /Users/matanya/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1
debug1: match: OpenSSH_5.5p1 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 140/256
debug2: bits set: 543/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 31:4c:71:e0:56:14:04:0d:c7:b2:6c:fc:8a:42:33:2e
debug1: Host 'ssh1.eu1.frbit.com' is known and matches the RSA host key.
debug1: Found key in /Users/matanya/.ssh/known_hosts:2
debug2: bits set: 513/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received

ssh-agent debugging

UPDATE: going through my local (ssh client machine) system.log I found the following:

Mar  6 10:28:17 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent[574]): Exited with code: 1
Mar  6 10:28:17 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent): Throttling respawn: Will start in 10 seconds
Mar  6 10:28:27 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent[575]): Exited with code: 1
Mar  6 10:28:27 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent): Throttling respawn: Will start in 10 seconds

What does Code 1 stand for?

UPDATE: I found the file that launchd has problems with at System/Library/LaunchAgents/org.openbsd.ssh-agent.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.openbsd.ssh-agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/ssh-agent</string>
        <string>-l</string>
    </array>
    <key>ServiceIPC</key>
    <true/>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
        <dict>
            <key>SecureSocketWithKey</key>
            <string>SSH_AUTH_SOCK</string>
        </dict>
    </dict>
        <key>EnableTransactions</key>
        <true/>
</dict>
</plist>

When I run /usr/bin/ssh-agent I get:

SSH_AUTH_SOCK=/var/folders/pg/1g6_hnwx47bgqv5vcm1lq18h0000gn/T//ssh-01WuaHF32SlV/agent.2145; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2146; export SSH_AGENT_PID;
echo Agent pid 2146;

as for the -l flag (<string>-l</string>) there is no such flag on my version of ssh-agent. Outputs:

ssh-agent: illegal option -- l

ps aux | grep ssh outputs:

matanya         1121   0.0  0.0  2441136   3280   ??  S     1:53PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1116   0.0  0.0  2441136   3280   ??  S     1:52PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1101   0.0  0.0  2441136   3280   ??  S     1:51PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1095   0.0  0.0  2441136   3280   ??  S     1:50PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1084   0.0  0.0  2441136   3280   ??  S     1:50PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1593   0.0  0.0  2439184   2092 s000  S+    2:36PM   0:00.00 grep ssh

SSH version: OpenSSH_5.8p2, OpenSSL 0.9.8r 8 Feb 2011

UPDATE: I've discovered that it doesn't matter with which user I initially login on system boot – be it my own or the root – ssh won't work until i explicitly switch user in the terminal (su -or su matanya)

UPDATE:

I checked the code signatures.
Ran: codesign -vv /usr/bin/ssh-agent:

received:

/usr/bin/ssh-agent: code object is not signed at all
In architecture: x86_64

Should be:

/usr/bin/ssh-agent: valid on disk
/usr/bin/ssh-agent: satisfies its Designated Requirement

UPDATE:

When I run :

eval `ssh-agent`
ssh-add

I can login with ssh.

Best Answer

Reason of the silent failing when connecting

Your system.log errors show you have an issue with your ssh-agent running locally on your iMac. For some reason it doesn't run even if launchd tries to restart it.

When you try to connect using any ssh client (CLI or Transmit) they try to use ssh-agent but they cannot connect to it as it's not running. Hence their waiting without output nor input.

I'm not sure what prevents your ssh-agent from running. However, to run your ssh client on the CLI and make it connect to your servers, you can try the following:

unset SSH_AUTH_SOCK
ssh u-indgo@ssh1.eu1.frbit.com
# (you'll then be asked for you pass phrase if you use one)

You can even try to launch Transmit from the same Terminal window:

open /Applications/Transmit.app

About ssh-agent debugging

If ssh-agent -l tells you the -l option is illegal, it means it's not the original Apple ssh-agent that your system is trying to run (-l is an Apple undocumented feature). The replacing ssh-agent is making launchd unhappy. This blog post might have some explanations why.

If you have third party ssh tools (coming from brew, macports or other channels), I'd recommend you move them out of the way or you upgrade them (provided they are launchd capable, i.e.: the -l option exists). A working ssh-agent invocation should answer something like:

antoine@amarante:~$ /usr/bin/ssh-agent -l
launch_msg: Operation not permitted

It is also a good idea to check you don't start ssh-agent from other places like .bashrc or other session startup scripts. Having multiple, and possibly different, ssh-agent running at the same time is potentially a source of problem.

Related Topic