Rsync backup keep deleted files

backuprsync

I have to backup a set of userdata with rsync. There is just one little thing I cant figure out how to do.
When a user deletes a file it is kept on the server. This is fine since it might have been deleted by mistake, but it would be nice if it did not stay there forever.

So how do I make rsync delete files on DEST when they have not been precent on SRC for a certain amount of time (say, 7 days)

I know that I could just write a script that manually uses rsync on each file in the folder and then determines if the file on DEST should be deleted, but it would be kind of dumb to do this just to learn later on that rsync has this feature already. I cant find it though.

Many thanks for you time

Best Answer

Rsync won't do this on its own. Something like rsnapshot will let you maintain multiple generations of rsync-based backups (hourly, daily, monthly, etc) such that you can retrieve an accidently-deleted file. By using hard links, rsnapshot lets you keep a number of snapshots laying around without using a lot more space than the "cost" of a full backup.

There are other solutions out there that do the same thing-- rsnapshot just happens to be one I have personal experience with. (I highly recommend using the "sync_first" option if you do decide to use it...)

Related Solutions

Chunking large rsync transfers

My solution to this was a different two-pass approach, where I trade off some disk space. I do rsync --only-write-batch on the server, then rsync the batch file itself to the destination, looping until the rsync succeeds. Once the batch is fully over rsync --read-batch on the destination recreates all the changes.

There are some unintended benefits to this for me as well:

because I'm more concerned that the backup "exists" than is "usable" I don't actually do the read-batch on the receiving end every day -- most of the time the batch is relatively small
I've been experimenting with --checksum-seed=1 ... I might be mis-reading the documentation but I think it makes the batch files more syncable (ie. when I don't do the --read-batch any given day, the next day's batch syncs faster because the previous day's batch is a good basis)
if the batch gets too big to send "in time" over the internet, I can sneaker-net it over on an external drive. By in-time I mean that if I can't get the batch over and read before the next day's backup starts.
although I don't personally do this, I could have two offsite backups in separate locations and send the batch to both of them.

Linux – How to Show Total Progress in Rsync

There is now an official way to do this in rsync (version 3.1.0 protocol version 31, tested with Ubuntu Trusty 14.04).

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for transferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

References:

Best Answer

Related Solutions

Chunking large rsync transfers

Linux – How to Show Total Progress in Rsync

Related Topic