I'm facing a rebuilding of the volume on which I host the mail storage (kerio mailserver, which uses maildirs). I need to backup and restore as quick as possible the 3.5+ millions (for about 600GB) small files of the store directory. It takes more than 12 hours via rsync to a NFS share, but I also have a 1TB firewire 800 raid1 disk that I can use (from some preliminary tests it's faster). I'm working off a XServe intel. What is the fastest way to do it? Rsync? Finder copy? tar?
Best way to backup and restore millions of files
rsync
Best Answer
I consider rsync to be one of the best options in this case. You may want to skip the Mac OS X built in rsync 2.6.9 and go for rsync 3 which is usually way faster. You can get a precompiled Mac optimized universal binary of rsync 3.0.6 with mlbackup, a free rsync based backup utility. (Installs rsync in /usr/local/maclemon/bin/rsync) For full and fair disclosure: I am the author of mlbackup.
If this server is your live mailserver, you may want to do an rsync copy to your FireWire device and jut let it take the time it needs to (and take time it will). When you're ready to go you just rerun the rsync command to have it updated to the current status, which will take significantly less time.
I'd suggest you just rsnyc to another drive, and run the mailserver from the new drive. That way you only need to copy once. Don't know if you have a spare bay left in your Xserve, or if you can run from the external drive for a few days. With this technique you can slit the backup and restore-to-internal-drive processes into to distinct events which you could probably do on two weekends to keep service downtime at a minimum.
Just make sure you have more than one backup of the data, just in case. You don't want to have anything go wrong.
You can also have a look at Alex's excellent mailbfr which will work up until 10.5.8, a Snow Leopard compatible version is in the works and should be released in the not too distant future.