EC2 Upload Speed – Drastic Drop for Single SCP Process

amazon ec2scpupload

I run batch compute jobs on EC2 instances (specifically i3.8xlarge) and upload large (10's GB) binary files to each instance. We have "gigabit" (Century Link) at the office and I typically upload (via scp) these files to EC2 at 30-40 MBps. The local machine is a RH7 Linux box. Often, however, I observe a curious thing: the upload speed drops precipitously to 300-400 kBps.

  • I verify (via speedtest.net) that the network is performing very adequately
  • The reduced upload speed is ALWAYS 300-400 kBps
  • The phenomenon started about a month ago
  • I can upload the same file, to the same instance (different output path, obviously), at the same time, from a different tab of the Gnome terminal, and I get good performance
  • The upload will typically resume a good rate unpredictably

So the bottleneck seems tied to a single scp process. Amazon doesn't advertise any type of upload throttling. Our network guys verify that the network is in good working order when I observe this.

Best Answer

I suspect your EBS Burst Balance has run out. That blog link gives you instructions on how to show your EBS volume burst credits.

The volume types page has a little information on EBS and bursting.

General Purpose SSD (gp2) volumes offer cost-effective storage that is ideal for a broad range of workloads. These volumes deliver single-digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Between a minimum of 100 IOPS (at 33.33 GiB and below) and a maximum of 16,000 IOPS (at 5,334 GiB and above), baseline performance scales linearly at 3 IOPS per GiB of volume size. AWS designs gp2 volumes to deliver their provisioned performance 99% of the time. A gp2 volume can range in size from 1 GiB to 16 TiB.

In short, you accumulate burst credits, and as you use your disk a lot you run out of credits. When that happens you get baseline performance. The larger the disk, the more credits, and the less likely you are to run out of credits. There are various articles (e.g. this one) you can read about them.

Instance store volumes do not have a burst system, which is why they work fast all the time.

The solution here is probably to buffer your writes to the instance store, then transfer them to the EBS volume in the background. rsync might be suitable.

Related Topic