Using torrent to share very large files

bittorrentfile-transfer

At our institute, we will soon need to routinely share large volumes of data (multi-terabyte range).

Would it make sense to use BitTorrent for this task?
How large of a CPU/memory overhead is to be expected, compared to common FTP servers?
Is it possible to achieve speeds matching a direct FTP transfer when copying from exactly one BitTorrent peer (the original storage server) to another?

Thank you very much.

Best Answer

I would think so. Be careful about the block-size you chose as it will need to be larger than standard for such a large amount of data
Not significant during the transfer, your bandwidth will be the bottleneck not your CPU. Generating the torrent meta-file (which involves hashing each block and the whole set of data) in the first place will take quite some time as will the final hash check after the transfer has completed on teh client
Yes. Unless your connectivity provider, the client's provider, or somewhere between, is selectively shaping P2P traffic.

To mitigate issues regarding points 1 and 2, if you can split the data into smaller chunks and have separate torrents for each chunk you might find the size of the data easier to handle.

Also note that you will need to regenerate the torrent metafiles if any data in the file(s) they cover is updated. If small parts of the data change without the rest changing, you probably find rsync to be a much more efficient solution.

How large are the files in the dataset and what is the spread like (several multi-gig files?, many smaller ones?, ...)?

Related Solutions

Linux – How to Copy Large Number of Files Quickly Between Servers

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh's comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir

Linux – Copying a large directory tree locally? cp or rsync

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Best Answer

Related Solutions

Linux – How to Copy Large Number of Files Quickly Between Servers

Linux – Copying a large directory tree locally? cp or rsync

Related Topic