MS DPM – Slow performance of consistency check

I'm guessing that you've got some sort of storage performance problem going on. If you run sp_who2 on the spid that's expanding the file and see what the wait type is.

Because EC2 is a shared hosting environment (your host shares the same hardware with other hosts), you can see substantial variability in I/O operations. EBS volumes are essentially NAS and share the same NIC with network traffic. Each physical host has only a 1Gb connection to the backbone. So, not only do you have contention with other customer's network operations, you also have network contention with their and your disks. In practice, the network contention is not ordinarily a problem unless you are sharing the box with many other high-traffic customers. You can get around some of that by using larger instances (larger instances take up a larger percentage of the box and thus have fewer shared resources).

What kind of iops are you experiencing at peak and during these problem periods? (sar -d tps column)

What is your steal time during these periods? (iostat -x 1 or sar -u).

You can increase your IOP capacity, which should help your iowait time, by software RAIDing multiple EBS volumes together. It sounds kludgy, but it actually works. This will not solve network contention problems, but with your traffic, I highly doubt you are saturating the link. It is possible that another customer is, however, and causing you some pain.

Sometimes, unfortunately, a simple solution to this type of problem is to simply respin the instance. It will likely come up on a different host with different shared customers. It is somewhat common practice for EC2 customers to spin instances, run some benchmarks, and respin if they are unhappy with the results.

Another recommendation is to split your web and database tiers into different servers. A single server with web/db is usually a bad idea for a large number of reasons and in this case is probably making it even more difficult to diagnose the bottleneck.