SSH key stopped working on Ubuntu Azure VPS

azurepublic-keysshubuntu-14.04

I have an Ubuntu instance that I'm running on Azure. I've been using it for the last 6 months or so to level up my novice Linux admin skills. Yesterday I was configuring a Capistrano deployment to this server when my ssh key stopped working all of a sudden. In the past I've seen Permission denied while trying to ssh to the box and I've just readded my private key to the key agent and it worked again. I was never really sure why the key stopped working but since I could fix it without too much hassle and it didn't occur very often I didn't think much of it. Yesterday, though, I wasn't able to fix the problem that way and I've been banging my head trying to figure out the problem since.

Here are the steps I've taken and the things I've tried:

  • Here is what I normally run (which has worked for months up until yesterday): ssh azure-southerneer

  • The relevant entry in my ~/.ssh/config:

    Host azure-southerneer
Hostname southerneer.cloudapp.net
User azureuser
IdentityFile ~/.ssh/azure-wp-eric.key
IdentitiesOnly yes
ServerAliveInterval 300 
StrictHostKeyChecking no
  • The output: Permission denied (publickey).

  • If I run it with the -v flag here is what I get:

    OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/kirkham/.ssh/config
debug1: /Users/kirkham/.ssh/config line 27: Applying options for azure-southerneer
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 102: Applying options for *
debug1: Connecting to southerneer.cloudapp.net [191.237.4.63] port 22.
debug1: Connection established.
debug1: identity file /Users/kirkham/.ssh/azure-wp-eric.key type -1
debug1: identity file /Users/kirkham/.ssh/azure-wp-eric.key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-8 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 77:6f:e6:84:ca:09:04:83:fd:37:96:3f:d3:2a:77:a6
debug1: Host 'southerneer.cloudapp.net' is known and matches the RSA host key.
debug1: Found key in /Users/kirkham/.ssh/known_hosts:78
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/kirkham/.ssh/azure-wp-eric.key
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

WTF?! It's trying the correct key, but it just doesn't like it anymore?!

  • I know that the server is up and running because the websites that I'm hosting there still run great. From the verbose ssh output it's obviously responding, it just doesn't like my azure-wp-eric.key (which hasn't changed).
  • I wasn't purposefully toying with permissions on my azureuser home directory on the server or with the .ssh directory, so that shouldn't have mucked up anything.
  • I've tried just bouncing the server from the Azure management portal, but that didn't solve anything.
  • I did run sudo apt-get update on the server yesterday morning, but I was able to ssh fine all through the day while testing my capistrano deployment so that doesn't seem to have caused the problem.

At this point I'm at my wit's end in terms of things to check. I'm obviously not very well versed at troubleshooting ssh problems and so I'm slowly working my way through articles trying to find what I'm missing. I don't know of any way to methodically figure out what I'm doing wrong which, as a developer, really bothers me. Is there something I'm overlooking? Are there other things to verify that I'm on the right or wrong track? Has anyone else experienced similar issues on Ubuntu or Azure VMs? Please let me know if there are any further details that I could supply to help diagnose the problem. Please help!

Best Answer

I realize my mistake. Thanks @kasperd for helping me figure it out. I followed Azure's SSH creation instructions to the letter. When I generated my key, I used this command

openssl req -x509 -key ~/.ssh/id_rsa -nodes -days 365 -newkey rsa:2048 -out myCert.pem

which generates a new key that expires after 365 days. It just so happens that yesterday was that 365th day which means my key no longer works. Since that was the only key to the VM I guess I'm screwed? Kinda seems like it. Ugh.

Update: Per @kasperd 's comments, the fact that my server stopped letting me SSH in exactly a year after setting it up may just be a crazy coincidence. Thanks to a conversation via the discussion thread over on the Azure documentation site, I discovered that Azure Linux VMs created via the gallery are loaded with the Microsoft Azure Linux Agent which allows admins to reset SSH through the command line (i.e. Powershell). So while it's a pain for me to switch over to Windows, I was able to follow the steps and reset SSH options on my VM. Unfortunately, I still wasn't able to SSH into my machine. Obviously I did something drastic to my machine that wasn't covered by the reset options, but for users with less drastic cases hopefully they can follow some of these links to help their situation.

Related Topic