EC2 Autoscaling AMI and auto attaching new EBD from snapshot

amazon ec2amazon-ebsautoscaling

Trying to work out how to configure EC2 autoscaling to automatically attach and EBS volume when a new instance is spawned.

Ideally we're trying to achieve the following when a new instances is spawned.

  1. Generate a snapshot from the master instance, one of the currently live instances in the group (this may vary as autoscaling adds and removes instances.)

  2. Create a new instance from our AMI

  3. Create an EBS volume from the snapshot in 1.

  4. Attach and mount the volume to the instance

  5. Add the new instance to the loadbalancer.

Is this possible. the key problem is the with handling EBS side of things…

Any advise

Best Answer

This is possible to a degree with some basic API calls. How to make the API calls is up to you, they can be done from the command line using the tools already in the path (or at least they should already be in there) or by using the various libraries Amazon has build SDK's for, including Java, Ruby, Python, PHP and .NET. The instructions below are targeted at a linux machine using bash scripts, although there are comparable Windows commands for them, I do not know the quirks related to their use.

I use AMI to refer to the Amazon Machine Image, and API call to refer to the command line tools, both those referred to as AMI tools, and API tools, since their naming is unique.

  1. To generate a snapshot from an EBS volume, the volume must be detached from the instance. To implement this in a production environment you will likely want to have a write only slave with some kind of operations log, so you can unmount it, snapshot it, remount the drive, and apply any missing operations to catch it up.

    • Unmount the instance from your machine, taking care to deal with any consequences in your application (like not having a database to write to). Normally I use umount -v /dev/sdf /$MOUNT_DIR. Use the -v switch to get some output for error handling.
    • Pass the volume-id for the volume to create the snapshot using ec2-create-snapshot. The drive does not need to be unmounted to do this, however it is safer if you do. Certain filesystems, like xfs, can freeze the filesystem, to allow for safe mounted snapshots. Be sure to save the snapshot ID for use later.
    • Remount the drive if necessary using mount -v -t [filesystem type] /dev/sdf /$MOUNT_DIR

  2. Bundling a new instance from an already deployed AMI is fairly straightforward. Ideally, your AMI is set up so that all mostly static information (configurations, libraries, installed programs etc) are on the ephemeral storage of the device, and all data (dbs, logs, etc) are stored on the attached drives. This is ideal, because the AMI bundling commands need to lock the FS, but do allow you to exclude certain directories. NOTE: For each of these you will need to provide your Amazon credentials, which spans several things from your account page including your Access Key ID, Secret Access Key, Cert, Private Key, and AWS Account ID.

    • Bundle your root file system using ec2-bundle-vol
    • Upload the bundle to Amazon using ec2-upload-bundle
    • Register the bundle with Amazon so you can launch an instance with it using ec2-register. Keep track of the new AMI ID for use later to relaunch.
    • To launch an image, either one you just created or one you would like to reuse, call ec2-run-instances, specify details as needed.

  3. Creating an EBS from a snapshot is easy with an API call.

    • Once you have the snapshot ID, just create your volume and specify the ID with ec2-create-volume. Be careful however, the volume must be the same size as the snapshot, if you need to increase the size, mount the larger drive at the same time and copy the data over in your favorite fashion.

  4. To attach and mount the drive, you need to do a combination of API calls on operating system commands.

    • Use ec2-create-volume and parse the return of the command for the Volume ID, so that you can call ec2-attach-volume. For the volume information, mount to the later device locations, as earlier ones are often in use for ephemeral storage (the instance itself). On linux, start with /dev/sdf or later. I have use /dev/sdf in this post, but as long as you are consistent, any lettering after f and up to p should work.
    • Mount the device in the file system. In linux, I use mount -v -t [File System Type] /dev/sdf /$MOUNT_DIR. The -v switch gives you some output you can parse for error handling.

  5. Again, another API call. I don't have a ton of experience with using this via automated scripts, but it should be as simple as calling elb-register-instances-with-lb.

Be sure to do error handling with the responses from these calls. A few months back I had particular trouble with the ec2-upload-bundle calls timing out and failing to finish uploading the entire image. If you decide to use an SDK, the steps should be mostly the same as all the Amazon API commands are included in the SDKs, albeit with different syntax.