Ubuntu – How to enforce rsync against performing checksum

backupchecksumrsyncUbuntu

We perform nightly backups of a Windows file server by first creating an incremental backup file on the server each night (as well as a complete backup on Thursday night), and then copying that to a backup server running Linux/Ubuntu. In order to maintain off-site redundancy, we then rsync the backup directory to an external drive which is rotated after each nightly run.

Over time, the number of incremental backup files has increased at a steady pace (although, the size of these files has varied).

We've also noticed that the rsync process has steadily been taking longer, even though it is only copying the latest two files to the external drive.

This is the command:

rsync -vr --delete-before --log-file=/rsync_log.csv /backup/archive/ /mnt/usbdisk/  2>&1

When we tested the command, and investigated the log file, we saw this:

...
2011/02/14 23:59:35 [14054] >f..T...... IncrementalBackup_2011_02_06_20.bkf
2011/02/15 00:00:45 [14054] >f..T...... IncrementalBackup_2011_02_08_20.bkf
2011/02/15 00:03:22 [14054] >f..T...... IncrementalBackup_2011_02_09_20.bkf
2011/02/15 00:04:36 [14054] >f..T...... IncrementalBackup_2011_02_11_20.bkf
2011/02/15 00:04:51 [14054] >f..T...... IncrementalBackup_2011_02_12_20.bkf
2011/02/15 00:05:06 [14054] >f..T...... IncrementalBackup_2011_02_13_20.bkf
2011/02/15 00:06:13 [14054] >f+++++++++ IncrementalBackup_2011_02_14_20.bkf
2011/02/15 00:54:32 [14054] >f..T...... Thursday_Full_Backup_2011_01_20.bkf
2011/02/15 03:24:41 [14054] >f..T...... Thursday_Full_Backup_2011_01_27.bkf
...

What we found was the time taken on each file related to the size of the file – even when skipping it (example – the full backup took about 2.5 hours to process, while the incrementals about 2-3 minutes or less).

The only file actually copied is the latest incremental file.

The only explanation we can think of is that rysnc is performing a checksum of the file – even though the documentation says it does not by default, and we have not specified the –checksum switch on the command. Surely it can't take 2.5 hours to determine the timestamp and filesize?

After having gone over the documentation, I can't find any other explanation than the checksum is being calculated. So, is there a way to be sure that checksum is disabled?

Best Answer

Maybe the problem is in the timestamps of the files. If backups are created using a windows program it can mess the timestamps. If I replicate your sample with a bunch of pictures I obtain this log

2011/02/15 03:46:46 [61820] delta-transmission disabled for local transfer or --whole-file
2011/02/15 03:46:46 [61820] .d..t....... ./
2011/02/15 03:46:46 [61820] IMG_0055.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0056.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0057.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0058.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0059.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0060.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0061.JPG is uptodate
2011/02/15 03:46:46 [61820] >f..t....... IMG_0062.JPG
2011/02/15 03:46:46 [61820] IMG_0063.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0064.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0065.JPG is uptodate
2011/02/15 03:46:46 [61820] IMG_0066.JPG is uptodate
2011/02/15 03:46:46 [61820] total: matches=0  hash_hits=0  false_alarms=0 data=5911343
2011/02/15 03:46:46 [61820] sent 5912367 bytes  received 67 bytes  11824868.00 bytes/sec
2011/02/15 03:46:46 [61820] total size is 75450221  speedup is 12.76

So the files are tranferred each time otherwise you should have the log message file xxx is uptodate. Try to increase rsync verbosity with more than one v flag to find out what's happening.

Related Topic