Fastest way to copy large file directory over slow internet

deduplicationfile-transfer

I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a small problem considering I need to copy about 85GB of data.

I used a file transfer calculator and it said it would take 11 days and some change.

I was wondering if there is any way to leverage something like DeDup to speed up the transfer.

The server I would be copying from is Windows 2003 R2, my box is Windows 7 Pro x64.

I was going to transfer the data to my iSCSI target which is running Ubuntu Linux, with ZFS and SCST. Would the dedup being performed by ZFS aid in the transfer so does the data need to be DeDup'd on the sending end?

Best Answer

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B.

Don't underestimate the power of the postal service.

Related Solutions

Sql-server – the simplest and fastest way to transfer large file through a Windows network

A failure predictable at 2Gb sounds like the target filesystem is to blame... Are both on NTFS? Are you piping through any compression (zip used to fail at 2gb boundaries) ((is apache doing compression))

I have copied many files over 20Gb using robocopy (as others have mentioned) but I'd avoid using the /MIR switch until you are sure you have got the copy doing what you want - since it will delete files as well as copy them.

SMB suffers from a one packet at a time limit so is often the slower way to copy files - you have the option to copy using push or pull. Personally, I prefer the push method (copy is initiated by the source).

Linux – How to Copy Large Number of Files Quickly Between Servers

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh's comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir

Best Answer

Related Solutions

Sql-server – the simplest and fastest way to transfer large file through a Windows network

Linux – How to Copy Large Number of Files Quickly Between Servers

Related Topic