What will happen to the EBS data when autoscaling the EC2 instance

amazon ec2amazon-ebsamazon-web-servicesautoscalingload balancing

I have various websites hosted on 1 same single EC2 instance so far. In the root EBS volume of such instance I have:
Codes and files of the websites which change every hour if not every minute
Apache and PHP configuration information which I change every one to two week for on-going development e.g. modifying the virtual hosts.

I would image autoscaling works the following way:

When traffic to my websites spikes and CPU ultilisation goes above 70% (my Cloud Watch settings), 1 more EC2 instances are spawned
These newly spawned EC2 will have latest codes & files in my original EC2 instance
During the time that the additional EC2 instances run, codes & files are kept consistent/ synchronised.
When CPU ultilisation goes below 40%, additional EC2 instances are stopped/ terminated (which ever accommodate 3. below)
When CPU ultilisation is above 70% again, additional EC2 instances are started (if they were stopped in 2.) or spawned (if they were terminated in 2.) with same requirements as 1.)

I've done searches and found tutorial such as this suggesting that I need to create an AMI first of all.
Wouldn't I lose data consistency because of such approach?
Can the AMI be created when traffic spike arises and can the data be kept consistent?

Best Answer

Your data on your first instance will not carry over to other instances. You will have to create a custom ami and bootstrap your instances if you want to autoscale. For example, you can keep a copy of your production files on S3 and have the instances look at S3 for the files on startup and periodically after startup. For files that are changing every minute, you may want to consider hosting them outside your instances, like on S3.

Related Solutions

EC2 Autoscaling AMI and auto attaching new EBD from snapshot

This is possible to a degree with some basic API calls. How to make the API calls is up to you, they can be done from the command line using the tools already in the path (or at least they should already be in there) or by using the various libraries Amazon has build SDK's for, including Java, Ruby, Python, PHP and .NET. The instructions below are targeted at a linux machine using bash scripts, although there are comparable Windows commands for them, I do not know the quirks related to their use.

I use AMI to refer to the Amazon Machine Image, and API call to refer to the command line tools, both those referred to as AMI tools, and API tools, since their naming is unique.

To generate a snapshot from an EBS volume, the volume must be detached from the instance. To implement this in a production environment you will likely want to have a write only slave with some kind of operations log, so you can unmount it, snapshot it, remount the drive, and apply any missing operations to catch it up.
- Unmount the instance from your machine, taking care to deal with any consequences in your application (like not having a database to write to). Normally I use umount -v /dev/sdf /$MOUNT_DIR. Use the -v switch to get some output for error handling.
- Pass the volume-id for the volume to create the snapshot using ec2-create-snapshot. The drive does not need to be unmounted to do this, however it is safer if you do. Certain filesystems, like xfs, can freeze the filesystem, to allow for safe mounted snapshots. Be sure to save the snapshot ID for use later.
- Remount the drive if necessary using mount -v -t [filesystem type] /dev/sdf /$MOUNT_DIR
Bundling a new instance from an already deployed AMI is fairly straightforward. Ideally, your AMI is set up so that all mostly static information (configurations, libraries, installed programs etc) are on the ephemeral storage of the device, and all data (dbs, logs, etc) are stored on the attached drives. This is ideal, because the AMI bundling commands need to lock the FS, but do allow you to exclude certain directories. NOTE: For each of these you will need to provide your Amazon credentials, which spans several things from your account page including your Access Key ID, Secret Access Key, Cert, Private Key, and AWS Account ID.
- Bundle your root file system using ec2-bundle-vol
- Upload the bundle to Amazon using ec2-upload-bundle
- Register the bundle with Amazon so you can launch an instance with it using ec2-register. Keep track of the new AMI ID for use later to relaunch.
- To launch an image, either one you just created or one you would like to reuse, call ec2-run-instances, specify details as needed.
Creating an EBS from a snapshot is easy with an API call.
- Once you have the snapshot ID, just create your volume and specify the ID with ec2-create-volume. Be careful however, the volume must be the same size as the snapshot, if you need to increase the size, mount the larger drive at the same time and copy the data over in your favorite fashion.
To attach and mount the drive, you need to do a combination of API calls on operating system commands.
- Use ec2-create-volume and parse the return of the command for the Volume ID, so that you can call ec2-attach-volume. For the volume information, mount to the later device locations, as earlier ones are often in use for ephemeral storage (the instance itself). On linux, start with /dev/sdf or later. I have use /dev/sdf in this post, but as long as you are consistent, any lettering after f and up to p should work.
- Mount the device in the file system. In linux, I use mount -v -t [File System Type] /dev/sdf /$MOUNT_DIR. The -v switch gives you some output you can parse for error handling.
Again, another API call. I don't have a ton of experience with using this via automated scripts, but it should be as simple as calling elb-register-instances-with-lb.

Be sure to do error handling with the responses from these calls. A few months back I had particular trouble with the ec2-upload-bundle calls timing out and failing to finish uploading the entire image. If you decide to use an SDK, the steps should be mostly the same as all the Amazon API commands are included in the SDKs, albeit with different syntax.

Ubuntu – Shutting an EC2 instance down locally (on the instance)

By default, when you shutdown/halt an EBS boot instance it is simply moved to the "stopped" state. This leave the instance available for restarting. You are no longer paying for instance hours, but you will continue paying a minimal charge for the EBS volume(s).

You can change this behavior so that a shutdown/halt will terminate the instance by setting the instance-initiated-shutdown-behavior to "terminate".

I've written a bit about this here:

Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data
http://alestic.com/2010/01/ec2-instance-locking

You'll want to run the instance with the option:

--instance-initiated-shutdown-behavior terminate

With this flag set, either of the following commands will terminate the EC2 instance:

sudo shutdown -h now
# or
sudo halt

You can also "Change Shutdown Behavior" through the AWS console for running instances (right mouse click).

Best Answer

Related Solutions

EC2 Autoscaling AMI and auto attaching new EBD from snapshot

Ubuntu – Shutting an EC2 instance down locally (on the instance)

Related Topic