Linux – Very High Network out in ec2 instance

Because EC2 is a shared hosting environment (your host shares the same hardware with other hosts), you can see substantial variability in I/O operations. EBS volumes are essentially NAS and share the same NIC with network traffic. Each physical host has only a 1Gb connection to the backbone. So, not only do you have contention with other customer's network operations, you also have network contention with their and your disks. In practice, the network contention is not ordinarily a problem unless you are sharing the box with many other high-traffic customers. You can get around some of that by using larger instances (larger instances take up a larger percentage of the box and thus have fewer shared resources).

What kind of iops are you experiencing at peak and during these problem periods? (sar -d tps column)

What is your steal time during these periods? (iostat -x 1 or sar -u).

You can increase your IOP capacity, which should help your iowait time, by software RAIDing multiple EBS volumes together. It sounds kludgy, but it actually works. This will not solve network contention problems, but with your traffic, I highly doubt you are saturating the link. It is possible that another customer is, however, and causing you some pain.

Sometimes, unfortunately, a simple solution to this type of problem is to simply respin the instance. It will likely come up on a different host with different shared customers. It is somewhat common practice for EC2 customers to spin instances, run some benchmarks, and respin if they are unhappy with the results.

Another recommendation is to split your web and database tiers into different servers. A single server with web/db is usually a bad idea for a large number of reasons and in this case is probably making it even more difficult to diagnose the bottleneck.

To clarify this point:

[Buffers are] used to temporarily store writes until they've been committed to disk. Is that a correct understanding?

No, that's not right.

You seem to understand the concept of cache memory. When a file is read from disk, the file is kept in memory as cache. If an application needs to access this file again, then access comes from RAM which is fast, as opposed to accessing the file again from disk, which is slow.

If an application needs to write to this file, then the write is performed on the file in RAM, which is fast, and the kernel marks those memory pages as "dirty". As far as the application's concerned, the write is complete and the app can get back to doing whatever it does.

The kernel handles flushing dirty pages out to disk later on. You can force a flush of all dirty pages with the sync command, or you'll see the flushing daemons (pdflush or bdflush) wake up from time to time.

You can see the amount of dirty memory at any time with cat /proc/meminfo | grep Dirty.

To correct your understanding, both clean pagecache (files which have been read) and dirty pagecache (files waiting to be written to disk) are counted as "cache" by Linux.

File cache can be freed if processes request more virtual memory allocations. Shared memory segments and tmpfs are also reported as "cache", but these cannot be freed like file cache can.

Usually "buffers" are memory allocations by running processes. Have a look in top -a or similar and see what process is taking up most of the RAM.