Home Backup Strategy for Large Amounts of Data

Are you sure that /home/.ecryptfs/A is locked for reading? I use ecryptfs and while I'm logged in and can browse and read the files in /home/.ecryptfs/myusername/.Private. I just tried going into that directory (and sub-directories) and opening files (using vim -b) and I could read them fine. I'd certainly want them locked for writing, but I don't see why they'd be locked for reading. What OS version are you using? (I'm on Ubuntu Lucid 10.04). Maybe ask a separate question about errors you're getting, because maybe something else is causing the problem.

To directly answer your question - back up the contents of /home/.ecryptfs/. This will backup (encrypted copies of) all the files for all users.

In addition you ought to be able to decrypt the files if necessary. So you should store the unwrapped passphrase somewhere secure, in case the user forgets their password, leaves ... To obtain it, have the user run

ecryptfs-unwrap-passphrase

while logged in, and store the result somewhere. It's small enough that you could write it down (double triple check it) and store it in a safe, or have two people keep half of it each or some such, depending how much security you require.

Otherwise you would need /home/.ecryptfs/*/.ecryptfs/wrapped-passphrase and the users' passwords.

You should also note that rsync won't be able to speed up file transfers when syncing encrypted data. Any change in the unencrypted file will completely change the encrypted file. And compression won't really work with encrypted data. This shouldn't be a big issue for your case, where the sync is across a LAN, but may be important for other people reading this question. Though rsync can still check if an encrypted file is unchanged, so it won't have to re-transfer unchanged files.

Readers of this question might also be interested in this guide to backing up by the maintainer of ecryptfs.

You need to take a step back and stop thinking "I've got 20TB on my NAS I need to back up!" and develop a storage strategy that takes into account the nature of your data:

• Where is it coming from and how much new data are you getting? (you've got this in your question)
• How is the data used once you have it? Are people editing the pictures? Do you keep the originals and generate edited versions?
• How long do you need to keep all the data? Are people still making changes to pictures from 2 years ago?

Depending on the answers to the last two questions, you probably need more of a Archiving System than a radically different backup system.

Data that is static (e.g. 2 year old pictures that you retain "just in case") doesn't need to be backed up every night, or even every week, it needs to be archived. What you actually do might be more complex, but conceptually, all the old pictures can be written off to tape (multiple copies!) and not backed up any more.

Based on your comments, some additional thoughts:

• Since you keep the originals of each shoot untouched and work on a copy, and assuming that at least some of the original pictures are duds, you might be able to cut the amount of data that needs to be backed up in half.

• If you still can't finish a full backup within whatever window of time you have, a common way to speed things up is to do a disk-to-disk backup first and then later copy the backup set off to tape.