Rsync of >4GB files

backupfileslimitationsrsync

silly one,

do you have any problems with rsync'ing large [ >4GB ] files under modern linux? [ 32bit, 64bit, large file support turned on ]? i've done some tests on my own between 2 64bit boxes and didn't have any problems transferring 6-10GB files. to make test thorough i altered files, run rsync again, checked md5… – all seems ok.

but after i saw this bug report i got a bit worried. i did some searching but have not found any confirmation of the problem.

thanks for your thoughts!

edit: file system: ext3, reiserfs

Best Answer

The error report that you've linked to does not seem to be a 4GB+ filesize related error. 429796854 bytes is just shy of 410Mb, and it seems to be a transport error rather than an rsync one. If

I would suspect that the transport connection (presumably SSH) has dropped, perhaps due to an inactivity timeout because the CPU spent so long between sends because it had to do something like checksum a very large file, and this is the reason that rsync reports a broken pipe.

I'm sure I've used rsync successfully on files over 4Gb in the past with 32bit clients and servers, and at least once where more then 4Gb was actually transferred rather than just considered for transfer.

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic