Ssh – Connection refused on Ubuntu EC2 after EBS expansion

amazon ec2connection-refusedssh

The problem

After starting an Ubuntu 14.04 EC2 instance with expanded HD volumes, ssh-ing into it fails with Connection refused.

The EBS expansion process

One of my Ubuntu 14.04 EC2 machines was running low on HD size. In order to solve the problem, I followed AWS's own manual on HD expansion:

  • Stopped the machine
  • Detached both volumes
  • Took a snapshot of both volumes
  • Created a larger volumes from the snapshot
  • Attached the new volumes and started the machine

In addition to that, I took the opportunity to add an Elastic IP to the machine, if that matters.

After starting the machine, I constantly get a Connection refused error when ssh-ing to it. I tried ssh from within the VPC to the private IP and from outside. I've used both the XX.X.XXX.XX IP and the ec2-XX-X-XXX-XX.compute-1.amazonaws.com DNS name, and both the original .pem key I've downloaded from AWS upon creation, and the ssh key I've placed in the .ssh/authorized_keys of the machine.

I get the same response:

ssh: connect to host ec2-XX-X-XXX-XX.compute-1.amazonaws.com port 22: Connection refused

Notes / What have I tried

  • I have connected the volumes to another instance and checked them. The files are there.
  • I tried removing from the PermitRootLogin lines from /etc/ssh/sshd_config.
  • I have tried connecting to the machine using the Java client in the EC2 console.
  • I have tried connecting the original volumes to the machine (before expansion). I still get Connection refused.
  • The root EBS volume is connected at /dev/sda1.

Solution

Update: Issue solved. Thanks a bunch to everybody here!

Best Answer

just putting this dirty hacky script, I used this to debug once (you just need python 2.xx installed on the machine). This is dirty but can help anyway!

Attach and mount your volume to another VM and create a file on it, eg /whatever_mount_path/you_like/cgi-bin/cmd.py ('cgi-bin' is important), with the content as below:

#!/usr/bin/env python
import os
br='<br/>'
print "Content-Type: text/html"
qs=os.environ['QUERY_STRING']
qs = {qs.split('=')[0]:qs.split('=')[1] for qs in   os.environ['QUERY_STRING'].split('&')}
print "cmd: ", qs ['cmd'], br*2
res = os.popen(qs['cmd']).read().replace('\n',br)
print res

then in the "/etc/rc.local" of your volume, add those lines:

cd /path_to_cgi-bin   #should be the path to your cgi-bin directory created above, without the mount path
nohup python -m CGIHTTPServer 8000 >> nohup.out 2>&1 &

reattach your volume to the initial machine, and boot it. From now, you can execute system commands from your web browser, by giving your cmd as GET query string. eg:

http://<YOUR VM IP>:8000/cgi-bin/cmd.py?cmd=netstat -ltpn | grep 22

which will be encoded by your browser as:

http://<YOUR VM IP>:8000/cgi-bin/cmd.py?cmd=netstat%20-ltpn%20|%20grep%2022

just make sure that the port 8000 is free, and that your security group is configured properly