We run a number of web apps that store a lot of local data in small xml files.
One part of our backup / recovery strategy is to produce a local mirror of the file system via a VPN to the hosting centre.
The VPN connection is only via a 12Mbps ADSL and whilst there are a lot of files and directories, the actual number of files that changes is quite small.
Although the bandwidth is probably an issue, I'm seeing results such as the output below. The robocopy /MIR took 5 hours to run yet only 30 mins to actually perform the copy.
Does anyone have any suggestions as to ways to improve this. The 5 hours is now bordering on too slow and if we can't find a way to speed this up then we're going to have to come up with a completely different solution.
Total Copied Skipped Mismatch FAILED Extras
Dirs : 17625 6618 11007 0 0 0
Files : 1112430 1223 1111207 0 0 0
Bytes : 57.451 g 192.25 m 57.263 g 0 0 0
Times : 5:01:23 0:35:55 0:00:00 4:25:27
Speed : 93509 Bytes/sec.
Speed : 5.350 MegaBytes/min.
Ended : Fri Apr 16 05:54:23 2010
Best Answer
Robocopy will have to enumerate all local and remote files first, to determine which ones need to be transferred. This is most likely what is taking the time.
What about if you reset the Archive file attribute following a successful backup:
Then every time a file is written to, the Archive bit will be automatically set. Next time, you can tell Robocopy to only archive files with the A flag set:
I haven't tested this, but I believe it should be quicker as Robocopy will have a lot fewer files to process.
Another idea would be to run a scheduled job on the remote server (if this is possible) to zip up the entire directory structure and then just copy the resulting zip file over the VPN. XML will compress nicely and copying a single file will be much more efficient over a high latency link.