Linux – Transfer between internal disks slows down for large files

benchmarklinuxperformanceubuntu-11.04

I'm running a Ubuntu Server 11.04 (amd64) on a machine with 1.6Ghz dual-core CPU, 4GB RAM, and 2x250GB 7200 SATA drives (both 3.0Gb/s and newly formatted from a fresh install). When I transfer relatively small files (~100MB) between the two drives, I achieve transfer rates that are (better) than expected, in the 300-350MB/sec range. However, transferring a 500MB file achieves about 50MB/sec, while a 1GB file achieves about 25MB/sec.

Here are some tests I ran that show this behavior:

/disk1 $ dd if=/dev/zero of=output-small.dat bs=1M count=128
 bytes (134 MB) copied, 0.388619 s, 345 MB/s
/disk1 $ dd if=/dev/zero of=output-med.dat bs=1M count=500
524288000 bytes (524 MB) copied, 20.0151 s, 26.2 MB/s
/disk1 $ dd if=/dev/zero of=output-large.dat bs=1M count=1000
1048576000 bytes (1.0 GB) copied, 45.7304 s, 22.9 MB/s

/disk2 $ dd if=/dev/zero of=output-small.dat bs=1M count=128
134217728 bytes (134 MB) copied, 0.372266 s, 361 MB/s
/disk2 $ dd if=/dev/zero of=output-med.dat bs=1M count=500
524288000 bytes (524 MB) copied, 10.0856 s, 52.0 MB/s
/disk2 $ dd if=/dev/zero of=output-large.dat bs=1M count=1000
1048576000 bytes (1.0 GB) copied, 37.1851 s, 28.2 MB/s

$ time cp /disk1/output-small.dat /disk2/test-small.dat
real    0m0.532s
user    0m0.000s
sys     0m0.410s

$ time cp output-med.dat /playpen/test-med.dat
real    0m30.879s
user    0m0.010s
sys     0m3.740s

$ time cp /disk1/output-large.dat /disk2/test-large.dat
real    0m26.643s
user    0m0.010s
sys     0m3.390s

Finally, when running the command pv < /disk1/output-large.dat > /disk2/test.dat, I initially see excellent transfer speeds, but then the rate drops off to the 1-10MB/sec range, periodically jumping (much) higher. I know this is not an particularly useful measurement of disk performance as I'm going through pv, but it is consistent performance with my other tests.

I have a couple questions:

  • What could be causing my transfer speeds to decrease with larger files? Shouldn't it be increasing?
  • What speeds should I expect to see with this kind of setup?
  • The time spent (presumably) waiting on IO (from time, real – sys) seems suspiciously high to me. Is it?

Best Answer

Small files fit in the page cache, thus the test finishes before stuff is on disk. Try adding the conv=fsync option to dd.

For the 1 GB file, perhaps you're running into some side-effects of writeback resulting in less throughput than the disks ought to be able to handle. You could try the dd options iflag=direct and oflag=direct to bypass the page cache and see how this affects the result.

Related Topic