Linux – EC2 EBS AMI Instance stopping/restarting doesn’t start services

This is possible to a degree with some basic API calls. How to make the API calls is up to you, they can be done from the command line using the tools already in the path (or at least they should already be in there) or by using the various libraries Amazon has build SDK's for, including Java, Ruby, Python, PHP and .NET. The instructions below are targeted at a linux machine using bash scripts, although there are comparable Windows commands for them, I do not know the quirks related to their use.

I use AMI to refer to the Amazon Machine Image, and API call to refer to the command line tools, both those referred to as AMI tools, and API tools, since their naming is unique.

1. To generate a snapshot from an EBS volume, the volume must be detached from the instance. To implement this in a production environment you will likely want to have a write only slave with some kind of operations log, so you can unmount it, snapshot it, remount the drive, and apply any missing operations to catch it up.

   • Unmount the instance from your machine, taking care to deal with any consequences in your application (like not having a database to write to). Normally I use umount -v /dev/sdf /$MOUNT_DIR. Use the -v switch to get some output for error handling.
   • Pass the volume-id for the volume to create the snapshot using ec2-create-snapshot. The drive does not need to be unmounted to do this, however it is safer if you do. Certain filesystems, like xfs, can freeze the filesystem, to allow for safe mounted snapshots. Be sure to save the snapshot ID for use later.
   • Remount the drive if necessary using mount -v -t [filesystem type] /dev/sdf /$MOUNT_DIR

2. Bundling a new instance from an already deployed AMI is fairly straightforward. Ideally, your AMI is set up so that all mostly static information (configurations, libraries, installed programs etc) are on the ephemeral storage of the device, and all data (dbs, logs, etc) are stored on the attached drives. This is ideal, because the AMI bundling commands need to lock the FS, but do allow you to exclude certain directories. NOTE: For each of these you will need to provide your Amazon credentials, which spans several things from your account page including your Access Key ID, Secret Access Key, Cert, Private Key, and AWS Account ID.

   • Bundle your root file system using ec2-bundle-vol
   • Upload the bundle to Amazon using ec2-upload-bundle
   • Register the bundle with Amazon so you can launch an instance with it using ec2-register. Keep track of the new AMI ID for use later to relaunch.
   • To launch an image, either one you just created or one you would like to reuse, call ec2-run-instances, specify details as needed.

3. Creating an EBS from a snapshot is easy with an API call.

   • Once you have the snapshot ID, just create your volume and specify the ID with ec2-create-volume. Be careful however, the volume must be the same size as the snapshot, if you need to increase the size, mount the larger drive at the same time and copy the data over in your favorite fashion.

4. To attach and mount the drive, you need to do a combination of API calls on operating system commands.

   • Use ec2-create-volume and parse the return of the command for the Volume ID, so that you can call ec2-attach-volume. For the volume information, mount to the later device locations, as earlier ones are often in use for ephemeral storage (the instance itself). On linux, start with /dev/sdf or later. I have use /dev/sdf in this post, but as long as you are consistent, any lettering after f and up to p should work.
   • Mount the device in the file system. In linux, I use mount -v -t [File System Type] /dev/sdf /$MOUNT_DIR. The -v switch gives you some output you can parse for error handling.

5. Again, another API call. I don't have a ton of experience with using this via automated scripts, but it should be as simple as calling elb-register-instances-with-lb.

Be sure to do error handling with the responses from these calls. A few months back I had particular trouble with the ec2-upload-bundle calls timing out and failing to finish uploading the entire image. If you decide to use an SDK, the steps should be mostly the same as all the Amazon API commands are included in the SDKs, albeit with different syntax.

An AMI, as you note, is a machine image. It's a total snapshot of a system stored as an image that can be launched as an instance. We'll get back to AMIs in a second.

Lets look at EBS. Your other two items are sub-items of this. EBS is a virtual block device. You can think of it as a hard drive, although it's really a bunch of software magic to link into another kind of storage device but make it look like a hard drive to an instance.

EBS is just the name for the whole service. Inside of EBS you have what are called volumes. These are the "unit" amazon is selling you. You create a volume and they allocate you X number of gigabytes and you use it like a hard drive that you can plug into any of your running computers (instances). Volumes can either be created blank or from a snapshot copy of previous volume, which brings us to the next topic.

Snapshots are ... well ... snapshots of volumes: an exact capture of what a volume looked like at a particular moment in time, including all its data. You could have a volume, attach it to your instance, fill it up with stuff, then snapshot it, but keep using it. The volume contents would keep changing as you used it as a file system but the snapshot would be frozen in time. You could create a new volume using this snapshot as a base. The new volume would look exactly like your first disk did when you took the snapshot. You could start using the new volume in place of the old one to roll-back your data, or maybe attach the same data set to a second machine. You can keep taking snapshots of volumes at any point in time. It's like a freeze-frame instance backup that can then easy be made into a new live disk (volume) whenever you need it.

So volumes can be based on new blank space or on a snapshot. Got that? Volumes can be attached and detached from any instances, but only connected to one instance at a time, just like the physical disk that they are a virtual abstraction of.

Now back to AMIs. These are tricky because there are two types. One creates an ephemeral instances where the root files system looks like a drive to the computer but actually sits in memory somewhere and vaporizes the minute it stops being used. The other kind is called an EBS backed instance. This means that when your instances loads up, it loads its root file system onto a new EBS volume, basically layering the EC2 virtual machine technology on top of their EBS technology. A regular EBS volume is something that sits next to EC2 and can be attached, but an EBS backed instance also IS a volume itself.

A regular AMI is just a big chunk of data that gets loaded up as a machine. An EBS backed AMI will get loaded up onto an EBS volume, so you can shut it down and it will start back up from where you left off just like a real disk would.

Now put it all together. If an instance is EBS backed, you can also snapshot it. Basically this does exactly what a regular snapshot would ... a freeze frame of the root disk of your computer at a moment in time. In practice, it does two things different. One is it shuts down your instance so that you get a copy of the disk as it would look to an OFF computer, not an ON one. This makes it easier to boot up :) So when you snapshot an instance, it shuts it down, takes the disk picture, then starts up again. Secondly, it saves that images as an AMI instead of as a regular disk snapshot. Basically it's a bootable snapshot of a volume.