How to keep MinIO backed up

We have ~1TB of data, and backup everything nightly using custom rsync scripts. The nice thing about rsync is that it only copies modified bytes (not the entire modified file) ... plus it compresses the data before transferring.

In our old system, we had to cart tapes and disks home since every day about 200GB of files were modified. But with rsync only the 1GB or so of modified data within these files are transmitted, and compressed down to ~200MB. As a result, we are able to backup everything to a remote site over a T1 in a few minutes (and under an hour on a very heavy maintenance day). The scripts also utilize Linux hard links to maintain 30 days of full archives (not incrementals) using only 2-4TB (before compression) of space. So we end up being able to restore archived data in seconds, while also maintaining off-site storage.

Luckily disk drive space has kept up with our company growth ... I think our total solution at both locations cost ~$1000.

Like anything in business, this comes down to requirements and cost-effectiveness. It depends(tm).

Here we go again! >smile< You'll end up with religious arguments in this post, if it goes the way that most of the posts about backup have on Server Fault.

You'll have the curmudgenly old guys like me who still generally recommend tape versus the trendy young guys who want to use disks like they were tape cartridges. Someone will bring up long-term retention and the longevity of tape, and someone else will chime in about how they have some IDE hard drives from 1992 that still work great.

After that, someone will mention the cost of tape media being less, per GB, than hard disk drives. Someone else will point to a weekly NewEgg special on 1TB hard disk drives and say that tape is more expensive. Someone else will factor in the cost of the tape drive and calculate the "break even" point for tape.

(No one usually argues for optical media, but I suppose there's a chance someone might.)

Personally, I wouldn't trust disks for long term archiving. You could use disks like tape cartridges (that Dell RD1000 that Russ Warren mentons is just 2.5" SATA drives inside a plastic enclosure that makes them seem "tape like" and, no doubt, is built to withstand some abuse), but you should think about the cost per media and the conditions in storage and transport.

Edit:

I've done a little spreadsheet (available at http://mx02.wellbury.com/misc/20090713-Server_Fault_Backup_Roundup.xls) that compares the following (with their calculated 1st year cost including drives):

• eSATA (500GB drives) - $1,300.00
• eSATA (1TB drives) - $1,950.00
• LTO-4 (internal drive, 1 tape / day) - $2,766.00
• LTO-4 (autoloader, 1 tape / day) - $4,566.00
• LTO-4 (autoloader, 2 tapes / day) - $5,632.00
• Dell RD1000 (1 500GB cartridge / day) - $16,224.00
• Dell RD1000 (2 500GB cartridges / day) - $31,199.00

I assumed a 5 day / week, 5 week rotation (35 days until a tape comes back around), running "full" backups with compression every day. I included the 500GB eSATA and RD1000 drives even though it was unclear if they'd actually hold the backup corpus or not.

I didn't factor in any kind of eSATA enclosures into my pricing. Realistically, there would need to be something surrounding the disks, but that's so subjective that I decided not to even bother. Handling those disks "bare" is asking for static electricity-induced damage to the circuit boards.

It's unclear what to say for a media replacement strategy. The SATA drives are warranted for 3 years (Hitachi), but I don't how they'd hold up to this kind of use. The LTO-4 tapes are lifetime warranted and typically good for 200 - 250 full passes (which would be over 19 years of use in this scenario). I have no idea what to say about media replacement on the RD1000's.

Those little 500GB 2.5" SATA drives in plastic boxes (aka RD1000 cartridges) at $599.00 ea. from Dell are a bit pricey, especially comapred to $50.00 500GB SATA drives or $41.00 LTO-4 tapes!