Ubuntu – Can’t mount after EBS snapshot and restore on Ubuntu 12.04

This is possible to a degree with some basic API calls. How to make the API calls is up to you, they can be done from the command line using the tools already in the path (or at least they should already be in there) or by using the various libraries Amazon has build SDK's for, including Java, Ruby, Python, PHP and .NET. The instructions below are targeted at a linux machine using bash scripts, although there are comparable Windows commands for them, I do not know the quirks related to their use.

I use AMI to refer to the Amazon Machine Image, and API call to refer to the command line tools, both those referred to as AMI tools, and API tools, since their naming is unique.

1. To generate a snapshot from an EBS volume, the volume must be detached from the instance. To implement this in a production environment you will likely want to have a write only slave with some kind of operations log, so you can unmount it, snapshot it, remount the drive, and apply any missing operations to catch it up.

   • Unmount the instance from your machine, taking care to deal with any consequences in your application (like not having a database to write to). Normally I use umount -v /dev/sdf /$MOUNT_DIR. Use the -v switch to get some output for error handling.
   • Pass the volume-id for the volume to create the snapshot using ec2-create-snapshot. The drive does not need to be unmounted to do this, however it is safer if you do. Certain filesystems, like xfs, can freeze the filesystem, to allow for safe mounted snapshots. Be sure to save the snapshot ID for use later.
   • Remount the drive if necessary using mount -v -t [filesystem type] /dev/sdf /$MOUNT_DIR

2. Bundling a new instance from an already deployed AMI is fairly straightforward. Ideally, your AMI is set up so that all mostly static information (configurations, libraries, installed programs etc) are on the ephemeral storage of the device, and all data (dbs, logs, etc) are stored on the attached drives. This is ideal, because the AMI bundling commands need to lock the FS, but do allow you to exclude certain directories. NOTE: For each of these you will need to provide your Amazon credentials, which spans several things from your account page including your Access Key ID, Secret Access Key, Cert, Private Key, and AWS Account ID.

   • Bundle your root file system using ec2-bundle-vol
   • Upload the bundle to Amazon using ec2-upload-bundle
   • Register the bundle with Amazon so you can launch an instance with it using ec2-register. Keep track of the new AMI ID for use later to relaunch.
   • To launch an image, either one you just created or one you would like to reuse, call ec2-run-instances, specify details as needed.

3. Creating an EBS from a snapshot is easy with an API call.

   • Once you have the snapshot ID, just create your volume and specify the ID with ec2-create-volume. Be careful however, the volume must be the same size as the snapshot, if you need to increase the size, mount the larger drive at the same time and copy the data over in your favorite fashion.

4. To attach and mount the drive, you need to do a combination of API calls on operating system commands.

   • Use ec2-create-volume and parse the return of the command for the Volume ID, so that you can call ec2-attach-volume. For the volume information, mount to the later device locations, as earlier ones are often in use for ephemeral storage (the instance itself). On linux, start with /dev/sdf or later. I have use /dev/sdf in this post, but as long as you are consistent, any lettering after f and up to p should work.
   • Mount the device in the file system. In linux, I use mount -v -t [File System Type] /dev/sdf /$MOUNT_DIR. The -v switch gives you some output you can parse for error handling.

5. Again, another API call. I don't have a ton of experience with using this via automated scripts, but it should be as simple as calling elb-register-instances-with-lb.

Be sure to do error handling with the responses from these calls. A few months back I had particular trouble with the ec2-upload-bundle calls timing out and failing to finish uploading the entire image. If you decide to use an SDK, the steps should be mostly the same as all the Amazon API commands are included in the SDKs, albeit with different syntax.

since you're striping data across the volumes, it would stand to reason that you have to put each NEW volume in the same location on the RAID as the volume from which the snapshot was created.

I tested your premise, and logical as it may seem, the observation is otherwise.

Let me detail this:
I have the exact same requirement as you do. However, the RAID0 that I am using has only 2 volumes.

I'm using Ubuntu 10 and have 2 EBS devices forming a RAID0 device formatted with XFS.

The raid0 device was creating using the following command:
sudo mdadm --create /dev/md0 --level 0 --metadata=1.1 --raid-devices 2 /dev/sdg /dev/sdh

I've installed MYSQL and a bunch of other software that are configured to use /dev/md0 to store their data files.

Using the same volumes: Once done, I umount everything, stop the Raid and reassemble it like so: sudo mdadm --assemble /dev/md0 /dev/sdh /dev/sdg The thing is that irrespective of the order of /dev/sdg /dev/sgh, the RAID reconstitutes itself correctly.

Using snapshots: Post this, I use ec2-consistent-snapshot to create snapshots of the 2 EBS disks together. I then create volumes from this disk, attach it to a new instance (that has been configured for the software already), reassemble the RAID (I've tried interchanging the order of the EBS volumes too), mount it and I'm ready to go.

Sounds strange, but it works.