What is the best way to backup millions of small files in a very small time period?
We have less than 5 hours to backup a file system which contains around 60 million files which are mostly small files.
We have tried several solutions such as richcopy, 7z, rsync and all of them seems to have a hard time. We are looking for the most optimal way…
We are open to putting the file in an archive first or transferring the file to another location via network or hard disk transfer
thanks
Best Answer
I'd strongly suggest using a file storage system that allows you to snapshot the volume and backup from the snapshot. This way there's little impact on the actual server and the backup system can take its time doing the backup without concern for the main system. You don't mention an operating system but something like ZFS or a NetApp filer would allow this and both are being used for this exact function all over the place. I'm sure there are other file systems that offer this but I know these work.
Hope this helps.