Linux – How to Copy Large Number of Files Quickly Between Servers

Small files fit in the page cache, thus the test finishes before stuff is on disk. Try adding the conv=fsync option to dd.

For the 1 GB file, perhaps you're running into some side-effects of writeback resulting in less throughput than the disks ought to be able to handle. You could try the dd options iflag=direct and oflag=direct to bypass the page cache and see how this affects the result.

I'd be tempted to answer "stop abusing the file system by treating it like a database" but I'm sure it wouldn't help you much ;)

First, you need to understand that if your limitation is in the bandwidth available on read, there isn't anything you can do to improve performance using a simple synch command. In such a case, you'll have to split the data when it's written either by changing the way the files are created (which means, as you guessed correctly, asking the devs to change the source program) or by using a product that does does geo-mirroring (like, for instance double-take: check around as I'm sure you'll find alternatives, that's just an example).

In similar cases, the main cause of problem isn't typically the file data but rather the meta-data access. Your first strategy will therefore be to divide the load into multiple process that act on (completely) different directories: that should help the file system keep up with providing you with the meta-data you need.

Another strategy is to use your backup system for that: replay your last incremental backups on the target to keep the database in sync.

Finally, there are more exotics strategies that can be applied in specific cases. For instance, I solved a similar problem on a Windows site by writing a program that loaded the files into the file system every few minutes, thus keeping the FS clean.