Windows Server 2003 R2 and later has support for DFSR, which I used extensively to sync and backup large amounts of data over a rather small pipe across three sites (80GB+ over a T1<-->T1<-->T1 topology).
msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx
Replicating data to multiple servers increases data availability and gives users in remote sites fast, reliable access to files. DFSR uses a new compression algorithm called Remote Differential Compression (RDC). RDC is a "diff over the wire" protocol that can be used to efficiently update files over a limited-bandwidth network. RDC detects insertions, removals, and rearrangements of data in files, enabling DFSR to replicate only the deltas (changes) when files are updated.
DFSR is fully multimaster and can be configured however you want. That will keep your data in sync on the "backup" location, for a very small amount of bandwidth and CPU. From here, you can use the Volume Shadow Copy Service.
technet.microsoft.com/en-us/library/cc785914.aspx
The Volume Shadow Copy Service can produce consistent shadow copies by coordinating with business applications, file-system services, backup applications, fast-recovery solutions, and storage hardware. Several features in the Windows Server 2003 operating systems use the Volume Shadow Copy Service, including Shadow Copies for Shared Folders and Backup.
The shadow copies reside on disk, and take "no space" aside from the changed files from snapshot to snapshot. This is a process that can run on a live dataset with no ill effects, aside from slightly increased disk I/O as the snapshot is being created.
I used this solution for quite some time with great success. Changes to files were written out to the other sites within seconds (even over the low bandwidth links), even in cases where just a few bytes out of a very large file changes. The snapshots can be accessed independently from any other snapshot taken at any point in time, which provides both backups in case of emergency and very very little overhead. I set the snapshots to fire at 5 hour intervals, in addition to once before the workday started, once during the lunch hour and once after the day was over.
With this, you could store all data in parallel at both locations, kept relatively up to date and "backed up" (which amounts to versioned, really) as often as you want it to.
The Shadow Copy Client can be installed on the client computers to give them access to the versioned files, too.
www.microsoft.com/downloads/details.aspx?FamilyId=E382358F-33C3-4DE7-ACD8-A33AC92D295E&displaylang=en
If a user accidentally deletes a file, they can right-click the folder, properties, Shadow Copies, select the latest snapshot and copy it out of the snapshot and into the live copy, right where it belongs.
MSSQL backups can be written out to a specific folder (or network share) which would then automatically be synched between sites and versioned on a schedule you define.
I've found that data redundancy and versioning with these can act as an awesome backup system. It also gives you the option to copy a specific snapshot offsite without interfering with the workflow, as the files it reads from aren't in use...
This should work with your setup, as the second backup site can be configured as a read-only sync/mirror.
Best Answer
Based on your comments it sounds like you've got a hierarchical storage management (HSM) system that is automatically handling restoring files from tape when they're accessed. (You don't specifically state this in your question, but your comment "...I believe it will attempt to call files from tape back to disk.")
The
-Wargument disables delta copying. This would have no effect on the problem you're trying to avoid.By default, rsync will use the modification timestamp and file size as a test to decide which files have changed. If your HSM maintains the file size and timestamp on the "stub" files (usually done via sparse files) then rsync shouldn't attempt to perform delta copies of the stubbed files. If, however, the sizes and timestamps don't match then you're going to assume the files have changed and attempt to do a copy.
I don't see any functionality in rsync that would allow for automatic exclusion of files that are stubbed out to your HSM. (I don't know what operating systems are at play, either. Windows, for example, has a file attribute that identifies files stubbed out to HSM.)
If your stub files don't have the proper timestamps and sizes then your best bet would probably be to generate an exclusion list of files that have been stubbed-out and use that to exclude files from rsync.