How to configure multiple hostfiles to split big backups in BackupPC

One approach would be to do it in two steps. 1st, rsync everything except the directory in question by using ignore patterns. 2nd, rsync just the directory using globbing in bash to get to the directory, like this:

rsync -av /usr/lib/mydata/bigdatadir*/ /mnt/usbvolume/bigdatadir/

Using a trailing slash on the source directory will effectively cause rsync to ignore the directory name, because it will be invoked on the contents of the directory rather than the directory itself. Of course, this globbing will be easiest if the directory is named with a constant prefix or suffix as in my example above. If it isn't, you could write a script to figure out the actual name of the directory, and do something more direct like this:

rsync -av /usr/lib/mydata/$BIGDATADIRNAME/ /mnt/usbvolume/bigdatadir/

In the end, your pseudo-code would be something like this:

1. Find $BIGDATADIRNAME
2. Rsync everything as you were before, but ignore $BIGDATADIRNAME
3. Rsync the contents of $BIGDATADIRNAME

After the first full backup, file transfers appear to be similar to those required for an incremental backup. More directory information is transfered. Incremental backups only pick up new and modified files since the last full backup. You need period full backups to track file and directory deletions. The transfer overheads are similar after the first full backup.

I just setup backuppc myself. Rsync has the nice feature that if both systems already have a file, it doesn't need to be transferred. There is a long period at the start of the backup where the information to determine which files need to be transferred is determined. The first full backup will transfer the whole file system. From my review of the documentation 1% of the files are transferred and compared to ensure pool is OK.

Rsync uses modification time to determine if the file has changed. It may use other criteria as well. Backuppc uses the same criteria.

Full backups build a full directory tree so file deletions are fully tracked. After the first full backup the transfer volumes are significantly lower. The backup speed seems similar to that of an Incremental backup.

Incremental backups don't record the information required to determine if a file has been deleted. They appear to have a relatively sparse directory tree sufficient to hold the new and modified files.

EDIT: I triggered a full backup on a system with only an incremental backup of most files. It ran very quickly and linked most files as same, rather than copying them. This is what I expected. Network load was significantly less than for the initial transfer.