I soon will have a folder with thousands of files, each file on the order of a few KB. I will need to transfer these across a Windows network from one UNC share to another. In general, is it faster to simply copy the files over en masse, or would it be faster to zip them up (e.g., using 7zip in fastest mode) and send one or a few large files? Or is there no difference in practice?
Windows – Which is Faster: Transferring Several Small Files or Few Large Files?
copyingfilesystemswindows
Related Solutions
There's a few ways to do this;
1.) DFS folder replication will work, there's a tutorial here and here's the MS webpage.
2.) Robocopy in mirroring mode will work as well, but it won't key off the fact that a file has changed. Here's a script that I shamelessly copied from somewhere, (why re-invent the wheel, eh? :) that I used when migrating large amounts of data from one share to another. (A big bonus with this is that it will do incremental mirroring, i.e. run it once to replicate, then run it again and it will only copy over the files that have changed.
Script is:
@ECHO OFF
SETLOCALSET _source="\workstation01\share"
SET _dest="\workstation02\share"
SET _what=/COPYALL /B /SEC /MIR
:: /COPYALL :: COPY ALL file info
:: /B :: copy files in Backup mode.
:: /SEC :: copy files with SECurity
:: /MIR :: MIRror a directory treeSET _options=/R:0 /W:0 /LOG:worstation_copy.txt /NFL /NDL
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries
:: /LOG :: Output log file
:: /NFL :: No file logging
:: /NDL :: No dir loggingROBOCOPY %_source% %_dest% %_what% %_options%
3.) If you use HP in a large corporate setting and have a vendor agreement or something set up with them, check into HP StorageWorks Storage Mirroring, however it's a bit heavy-handed for two windows folders.
4.) Finally, there's also, (beta warning! Beta warning!) Windows Live Mesh, but it's.. You know, beta. :)
Personally, I'd go with the robocopy option.
Best Answer
It is faster to transfer a single large file instead of lots of little files because of the overhead of negotiating the transfer. The negotiation is done for each file, so transferring a single file it needs to be done once, transferring n files means it needs to be done n times.
You will save yourself a lot of time if you zip first before the transfer.