Linux – EC2 SSH / SFTP issues

I have done a lot of searching on this issue, and can not find anything particularly relevant.

I have many instances spread across zones and grouped into various load balancers. One of the groups is an "API" utilized by various different applications for common and secure tasks.

In the API group we have one instance that uses an Elastic IP so that we can reliably use crons for tasks such as rsync etc. Someone before my time also decided it would be a good idea to hard code this IP into various applications in typical spaghetti code if else forest fashion… long story short, very important instance.

Two days ago I was suddenly unable to SSH to this guy (It was created over a year ago). The pem key works across ALL other instances, which have been around for relatively the same amount of time.

This is output from failed connection on home machine (yesterday) attempts using ssh -v -i path/file.pem user@ip

ssh -v -i <path>/<file>.pem <user>@<ip>                                                                                             
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011                                                        
debug1: Reading configuration data /etc/ssh/ssh_config                                                      
debug1: Applying options for *                                                                                      
debug1: Connecting to <ip> [<ip>] port 22.                                              
debug1: Connection established.                                                                                     
debug1: identity file /home/<user>/.ssh/id_rsa type -1                                                          
debug1: identity file /home/<user>/.ssh/id_rsa-cert type -1                                                 
debug1: identity file /home/<user>/.ssh/id_dsa type -1                                                          
debug1: identity file /home/<user>/.ssh/id_dsa-cert type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:78
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/<file>.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/<user>/.ssh/id_rsa
debug1: Trying private key: /home/<user>/.ssh/id_dsa
debug1: Trying private key: /home/<user>/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

Finally after searching for hours on what to do about this I ran ssh -Tvvv -i path/file.pem user@ip and magically I had a debug connection to the instance.

This was done from a machine at home. I killed the connection, and connected normally. No problems. Connected from a server at the office with no problem.

Today I can not SSH into the instance from my machine at the office, and all converted keys do not work on this instance either (i.e. ppk keys for filezilla etc.)

Here is -v debug output from a working machine (home machine)

ssh -v -i <path>/<file>.pem <user>@<ip>
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to <ip> [<ip>] port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/<file>.pem type -1
debug1: identity file /home/<user>/.ssh/<file>.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:78
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/<file>.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/<user>/.ssh/<file>.pem
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to <ip> ([<ip>]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Remote: Ignored authorized keys: bad ownership or modes for directory /home/<user>
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Welcome to blah...

All permissions on keys are identical across instances and machines connecting to instances.

Does anyone have any idea what can be done to correct this?

Thanks,
-David

–EDIT–

This is the -v debug output of work machine that still will not establish a SSH connection.

I also recently cleared the known hosts on this machine.

ssh -v -i <path>/<file>.pem <user>@<ip>
OpenSSH_6.0p1 Debian-3ubuntu1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to <ip> [<ip>] port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/<file>.pem type -1
debug1: identity file /home/<user>/.ssh/<file>.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 75:43:6d:03:43:f5:89:fa:8d:fe:64:e1:39:9a:73:26
debug1: Host '<ip>' is known and matches the ECDSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/<user>/.ssh/<file>.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

I would like to reiterate that all converted keys are not working for sftp despite whether an ssh connection can be established. Specifically .ppk key converted to be used in filezilla. The same follows that the .ppk works on all other instances except the problem child… and also worked previously.

–EDIT 2–

The machine I was able to connect with was restarted, and I can no longer connect with it.