Rsync huge dataset of small files 5TB, +M small files

copyrsynctransfer

I encountered a situation where an app server misconfig led to creation of around 5TB datasets where each dir contains huge number of small files.
We are in the process of transferring the files and change the application but the rsync fails on transferring the data. It is fails even locally between the local drives. I managed to copy only 3.5G overnight!
I tried to change the rsync switches and still no luck.
Here is what currently running on the server without any progress indication:
rsync -avhWc –no-compress –progress source destination
Some suggested the gigasync but the github and the site is unavailable.
Can anybody suggest a method to transfer the files?
Appreciate anyhelp

Best Answer

Try xargs+rsync:

 find . -type f -print0 | xargs -J % -0 rsync -aP % user@host:some/dir/

You can control how many files to pass as source to each call of rsync with -n E.g. to copy 200 files at every rsync:

 find . -type f -print0 | xargs -n 200 -J % -0 rsync -aP % user@host:some/dir/

If it's too slow you can run multiple copies of rsync in parallel with the -P option:

find . -type f -print0 | xargs -P 8 -n 200 -J % -0 rsync -aP % user@host:some/dir/

This will start 8 copies of rsync in parallel.

Related Solutions

How to Use Rsync Over FTP

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

How to Use Rsync Over FTP

Linux – Copying a large directory tree locally? cp or rsync

Related Topic