All sources are on windows OS, and destination backup is on Unix system (we are using Samba).
My source repository is similar to :
-Repository
--Folder1
---Files
--Folder2
---Files
etc...
I would like to get a destination similar to :
-Repository
--Folder1.zip
--Folder2.zip
etc...
Few things you should know :
– I have 30Tb of data to backup,
– I'm excluding folders older than 10 days (is MAXAGE:N can work with folders ?),
Is there a way to sync via Robocopy folders and zip files without need to zip source first ?
Workflow is actually :
– sync folders and files between source and destination via robocopy,
– zip destination folders,
– sync again (but how to compare ?, files in destination are now in zip format).
Best Answer
If you zip up a set of files, then modify one of the files and make another zip file you can't do a differential copy between the two, you will have to copy the entire zip over again. This is going to be a killer on your network, and on the source storage and server where you are continually re-reading and re-writing these files.
30TB of data - assuming you get a decent 50% compression will take about 70 hours to copy that over a Gigabit ethernet link assuming you are going to be able to realize about 60Meg/sec transfer rates for the entire period of time (50% utilization of the network on average). If you have a 10GigE link then you can expect to get 5-6 times that rate, maybe more if they are very good 10Gig links.
Depending on the source storage system and how many and how big the source files are you may also have problems reading and compressing them at that sort of rate. Likewise you'll need a reasonable target system to be able to support that steady 60Meg/sec write speed. And again scale that up if you are leveraging 10Gig and want to be able to sustain the transfer rates.
This seems like an ideal use case for in-situ compression or de-duplication on the target storage system to me - have you looked at any solutions that will do that for you?