How to troubleshoot slow performance on AWS EC2/RDS

Running everything on a single EC2 instance defeats the whole point of a cloud-based deployment: the ability to autoscale and self-heal. As I've written before, autoscaling is the heart and soul of AWS and if you're not using it, you'd be better off using a traditional co-lo server or VPS. It would be both cheaper and more durable.

I just completed an AWS deployment that is very similar to your needs. The client runs three, fairly high-volume Wordpress sites (quite a bit more traffic than yours). The config looks like this:

• Everything in a VPC for additional security. VPC contains six subnets across two availability zones (AZ)
• An Elastic Load Balancer that spans two subnets/AZs
• An autoscaling (AS) group of m1.small instances serve as the application tier. There are a min of two app servers and a max of 10, depending on the traffic load. In normal operation, it runs with just two and average CPU utilization is consistently under 15%. The AS group scales up and down with two instances at a time, one in each AZ.
• Each app instance runs Nginx + PHP-FPM + APC. In front of this stack I have Varnish installed to provide additional caching.
• A small, multi-AZ RDS instance in the two, private subnets. Even under extremely high traffic load, it's barely touched due to the amount of application-side caching.
• Static assets are served off of Cloudfront to reduce the load on the application servers even further.
• Files are stored on a pair of mirroring Gluster nodes, each in a separate AZ for HA purposes. A floating elastic IP address is assigned to one of the nodes, which gives you SSH or SFTP access into the actual Wordpress files. Nodes are part of it's own AS group, so if the fileserver were to die, it would be automatically killed and recreated. On boot, it reattaches the volume containing the Wordpress files. Automatic backups occur via a series of hourly volume snapshots.
• A t1.micro instance serves as a NAT gateway for instances in the private subnets.

This design does not have any single point-of-failure*, will automatically self-heal if an instance dies, and is smart enough to scale up or down according to resource demand. Total cost: about $200/mo. if you opt to purchase 1-year reserved instances.

I'm working on putting this configuration into a combination of a CloudFormation template and cloud-init/Python scripts that are automatically pulled from Github on boot. Basically it will allow anyone to pretty much push a button, wait for about an hour, and then come back and this whole environment will be waiting. I hope to have this completed by the end of the year. If you'd be interested in getting a copy of the template, send me an email to "jamie" at the website listed in my profile.

_{* The NAT instance is a SPoF, but that's primarily a design limitation of VPC. And it's a non-critical component that could fail and not affect the application.}

Do non-"PIOPS Optimized" instances still benefit from Provisioned IOPS?

Yes. You benefit from the PIOPS volumes being SSDs under the hood. Optimized instances have dedicated network to the EBS servers, so they get consistent performance. Non-optimized instances interact with EBS along with the rest of their network traffic, so heavy bandwidth use on the physical host can slow you down.

http://aws.typepad.com/aws/2012/08/fast-forward-provisioned-iops-ebs.html

As a point of reference, a standard EBS volume will generally provide about 100 IOPS on average, with the ability to burst to hundreds of IOPS on a best-effort basis.

With this in mind, I suspect you'll see performance improvements from PIOPS on the RDS, as you're averaging 50-100 writes and 200-400 reads on the RDS volume - you're likely in that best-effort burst state most of the time.

Your web server is doing zero IOPS from the charts, so I'd guess everything's already living happily in RAM and not hitting the disk. Do check that it doesn't have a secondary volume you should be looking at instead, though.