Samba – Robocopy mirror to Samba keeps rewriting files

Lessons Learned:

1. "User Account Control" is still of the devil. It was causing access denied for domain admin to read the user files.

2. Once UAC was disabled, I was still getting access denied. This turned out to be due to server 2008's use of JUNCTION POINTS in the user profile directories. This made an recursive loop like this:

C:\users\username\appdata\appdata\appdata\appdata\appdata...

So, when using robocopy vs. a 2008 server apparently /XJD and /XJF are pretty important. (Those are for "exclude junction points for files/directory").

After addressing both above points, no more problems.

If you zip up a set of files, then modify one of the files and make another zip file you can't do a differential copy between the two, you will have to copy the entire zip over again. This is going to be a killer on your network, and on the source storage and server where you are continually re-reading and re-writing these files.

30TB of data - assuming you get a decent 50% compression will take about 70 hours to copy that over a Gigabit ethernet link assuming you are going to be able to realize about 60Meg/sec transfer rates for the entire period of time (50% utilization of the network on average). If you have a 10GigE link then you can expect to get 5-6 times that rate, maybe more if they are very good 10Gig links.

Depending on the source storage system and how many and how big the source files are you may also have problems reading and compressing them at that sort of rate. Likewise you'll need a reasonable target system to be able to support that steady 60Meg/sec write speed. And again scale that up if you are leveraging 10Gig and want to be able to sustain the transfer rates.

This seems like an ideal use case for in-situ compression or de-duplication on the target storage system to me - have you looked at any solutions that will do that for you?