Here is the scenario:
I have 5 TB (yes, that's a T) of files on a Windows server that I need to migrate to a new server in as short and efficient time as possible. (Think: Robocopy, Rsync, etc as I plan to use differentials to do this over time). The files are in ~41,000 directories under a single parent directory (d:\files\folder1, d:\files\folder2, etc).
Since these are migrating to a new server, I want to split this up so that they are not all in the same "files" directory, but are instead split as logically as possible across multiple drive (trying to stay at about 2TB drive sizes for backup and replication purposes).
Robocopy doesn't have a regex option. Rsync would require a linux server, which isn't impossible, but adds overhead: this is a Windows to Windows move. I've found a way to loop through the 41,000 directories using powershell and initiate Robocopy for each directory individually, thus allowing me to specify destination…but this seems inefficient.
One other option I've considered is migrating everything at once, and then scripting out a copy to the other drives as needed. This would mean copying 2/3 of the files twice.
Have I missed anything obvious?
Best Answer
I've done window 2 windows rsync using the CygWin framework a couple of years back. Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.