After a routine reboot of a Linux ec2 instance with 1 small root volume and 1 small attached volume I was not able to ssh into the instance. It is not clear why rebooting the instance caused it to be inaccessible from ssh. The instance showed as running in the AWS console but ssh, http,etc was not responsive. I tried to create an AMI from this running instance. However, the AMI was never created. Instead I just saw "pending" in the AMI section of the AWS console for hours. Eventually I de-registered the AMI. Next I tried to stop the ec2 instance. However I am not able to stop the instance – it has been stuck in the stopping state for hours. I also tried force stopping the instance with no success. I then tried to detach the volumes but they constantly report "detaching" Does anyone have any suggestions about how to handle this? It seems that Amazon does not offer any kind of email or phone support unless you are a premium member. Thanks very much for your help.
EC2 Instance will not stop or terminate after trying to create an AMI from the running instance
amazon ec2
Related Solutions
Add --force option to ec2-detach-volume utility:
sudo apt-get install ec2-api-tools
ec2-detach-volume [LOGIN-OPTIONS] --force VOL-ID
An AMI, as you note, is a machine image. It's a total snapshot of a system stored as an image that can be launched as an instance. We'll get back to AMIs in a second.
Lets look at EBS. Your other two items are sub-items of this. EBS is a virtual block device. You can think of it as a hard drive, although it's really a bunch of software magic to link into another kind of storage device but make it look like a hard drive to an instance.
EBS is just the name for the whole service. Inside of EBS you have what are called volumes. These are the "unit" amazon is selling you. You create a volume and they allocate you X number of gigabytes and you use it like a hard drive that you can plug into any of your running computers (instances). Volumes can either be created blank or from a snapshot copy of previous volume, which brings us to the next topic.
Snapshots are ... well ... snapshots of volumes: an exact capture of what a volume looked like at a particular moment in time, including all its data. You could have a volume, attach it to your instance, fill it up with stuff, then snapshot it, but keep using it. The volume contents would keep changing as you used it as a file system but the snapshot would be frozen in time. You could create a new volume using this snapshot as a base. The new volume would look exactly like your first disk did when you took the snapshot. You could start using the new volume in place of the old one to roll-back your data, or maybe attach the same data set to a second machine. You can keep taking snapshots of volumes at any point in time. It's like a freeze-frame instance backup that can then easy be made into a new live disk (volume) whenever you need it.
So volumes can be based on new blank space or on a snapshot. Got that? Volumes can be attached and detached from any instances, but only connected to one instance at a time, just like the physical disk that they are a virtual abstraction of.
Now back to AMIs. These are tricky because there are two types. One creates an ephemeral instances where the root files system looks like a drive to the computer but actually sits in memory somewhere and vaporizes the minute it stops being used. The other kind is called an EBS backed instance. This means that when your instances loads up, it loads its root file system onto a new EBS volume, basically layering the EC2 virtual machine technology on top of their EBS technology. A regular EBS volume is something that sits next to EC2 and can be attached, but an EBS backed instance also IS a volume itself.
A regular AMI is just a big chunk of data that gets loaded up as a machine. An EBS backed AMI will get loaded up onto an EBS volume, so you can shut it down and it will start back up from where you left off just like a real disk would.
Now put it all together. If an instance is EBS backed, you can also snapshot it. Basically this does exactly what a regular snapshot would ... a freeze frame of the root disk of your computer at a moment in time. In practice, it does two things different. One is it shuts down your instance so that you get a copy of the disk as it would look to an OFF computer, not an ON one. This makes it easier to boot up :) So when you snapshot an instance, it shuts it down, takes the disk picture, then starts up again. Secondly, it saves that images as an AMI instead of as a regular disk snapshot. Basically it's a bootable snapshot of a volume.
Best Answer
If you don't have premium support, then "stuck" instances (where you can't stop/terminate them) and "stuck" volumes (where you can't detach/delete them) can be reported to Amazon on the EC2 forum:
Make sure you list the specific instance/volume ids involved.
Nobody but Amazon can really help in these situations.
Fortunately, you should not be charged for instance hours once it enters the "stopping" or "terminating" state.
Your original problem about not being able to connect to the instance through ssh could also be posted on the EC2 forum for help from the community, but it is a common problem and has many potential causes. I've written an article to help start diagnose this and to point out pieces of information you should include in your forum post:
I've also written an article describing a method that can be used to diagnose EBS boot instances by looking at the log files on the disk even if you can't connect to the instance:
However, you won't be able to use this approach until Amazon helps you detach your EBS volume.