Linux NTFS to NTFS rsync repeatedly recopying files

linuxntfsrsync

I wish to backup data mounted on an NTFS partition of a linux machine to the NTFS partition of another linux machine. The local NTFS drive is mounted with all files owned by root.root, but the remote NTFS drive is mounted using plugdev (so file ownership is root.plugdev).

rsync can apparently not copy owner/group and permission info in this situation, so I have been trying to run

sudo rsync -rltDvO --delete /ntfs/src/ user@remote.example.com:/ntfs/dest/

However, every time I run this command, it re-copies ALL the files, rather than just transferring the changed files. I have tried using the –modify-window flag, but that hasn't helped either.

Update: it is not recopying all the files, rather it is recalculating the checksums (or doing some other processing, causing the filename to be printed) for each file. This is opposed to what happens between to e2fs drives, where no filenames are printed if no changes have been made to the source and destination since the last rsync. I am guessing this processing is because the timestamps are not available on NTFS(?)

Next update: Permissions certainly seem to be a part of it: first, also using -u as an rsync flag stopped the processing of all the files on the destination, but this is not what I want. Secondly, if I use an account with root access (and rsync flags -avz) on the remote machine, I can actually change the timestamps on files and directories. However, some files continue to be recopied, and the –modify-window seems to help for that somewhat (but not completely, it seems – I'm still experimenting to find out why.)

Best Answer

As per my second update, the permissions of my user did not allow me to set times. Running the rsync using the root account on the destination machine, along with --modify-window=5 gave me suitable behaviour. (If you don't have root access on the destination, the -u flag could help, but only if you're sure no-one will change the files on the remote system.)

Related Solutions

Linux – Showing total progress in rsync: is it possible

There is now an official way to do this in rsync (version 3.1.0 protocol version 31, tested with Ubuntu Trusty 14.04).

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for transferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

References:

Linux – rsynced files not getting proper ACL

After a little help from my friends and much gnashing of teeth, I figured this out. Of course, the answer was in the man pages.

The problem stems from the fact that (from the acl man page): "modification of the file permission bits result in the modification of the associated ACL entries" and vice versa. Because without the proper flag, rsync uses the source permissions this changes the ACLs on the target.

The solution in rsync is to use the --chmod option. From the rsync man page (in the --perms section):

To give new files the destination-default permissions (while leaving existing files unchanged), make sure that the --perms option is off and use --chmod=ugo=rwX (which ensures that all non-masked bits get enabled).

However, the rights one would want to use depend on the intended target rights. As an example, in my case, the source perms were 700, with no ACL. The destination sub-directory had a default ACL as follows: d:u:user1:rwx,d:u:user2:rwx,u:user1:rwx,u:user2:rwx,d:m:rwx,m:rwx. That is, it has two named users, an explicit mask, as well as identical defaults. This equates to 770.

When using rsync (or cp-ing), the source permission of 700 override the default ACL mask, setting it to --- instead of rwx.

Since this uses case needs "user" and "group" but no "other" source permissions, I used the following flag in rsync: --chmod=ug=rwx. The letters after the first equal sign indicate that the following perms apply to the u(ser) and g(roup). The letters following the second equal sign indicate the permissions those users receive. This makes rsync set the perms on the target to 770.

The actual command is: rsync -av --chmod=ug=rwX /<source> /<destination>

A few things to note. First, rsync permission behaviour depends upon the version on the receiving end. Check man pages for the definitive answer. Second, the order of flags is important: -a, -A and so on can, in fact, be used but MUST come before the --chmod flag.

Lastly, I see no similar flag available for cp making rsync the only game in town for syncing files AND applying default ACLs on the target to files with different source permissions.

Best Answer

Related Solutions

Linux – Showing total progress in rsync: is it possible

Linux – rsynced files not getting proper ACL

Related Topic