Estimating the time needed to copy a big file

cpscp

I have to copy a relatively big file (6GB) to another server. To enforce consistency, I have to 'lock' this file and not use it during the transfer. I obviously want to minimize this downtime, but before this, I also want to estimate the time it will take to do so.

I can see 2 solutions :

copy the file locally and then transfer over the network (local). I'm thinking this will lead to the smallest downtime, but then, maybe I'm wrong as the disk will have to write and read the same file at the same time. Even with this, I have no idea how to estimate the time it will take to copy the file locally. Maybe creating a 6GB file and copying it would be meaningful, but then, how can I do this?
copy the file over the network directly (scp). This seams slow as network access is orders of magnitude slower than disk writing, but again, maybe I'm wrong.

Generally, how can I benchmark both approaches and decide which one is best for me?

Best Answer

Honestly... there's several ways to attack this. I have a few servers where I need to copy files that are in constant use... so I schedule a "maintenance window" of a few minutes where I can lock the files... create a snapshot (using dm-snapshot), bring the server back online and then do whatever with the snapshot'ed copy of the file. This allows me to have a stable version of the file... and bring the service back online with minimal down-time. Afterwards... I can remove the snapshot without affecting the live server. In your case... you can copy the 6gb snapshot'd file at your own pace as-needed.... without worry about the live-version of the file messing up the copy process.

http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

Related Solutions

Linux – Copy large file from one Linux server to another

Sneakernet Anyone?

Assuming this is a one time copy, I don't suppose its possible to just copy the file to a CD (or other media) and overnight it to the destination is there?

That might actually be your fastest option as a file transfer of that size, over that connection, might not copy correctly... in which case you get to start all over again.

rsync

My second choice/attempt would be rsync as it detects failed transfers, partial transfers, etc. and can pick up from where it left off.

rsync --progress file1 file2 user@remotemachine:/destination/directory

The --progress flag will give you some feedback instead of just sitting there and leaving you to second guess yourself. :-)

Vuze (bittorrent)

Third choice would probably be to try and use Vuze as a torrent server and then have your remote location use a standard bitorrent client to download it. I know of others who have done this but you know... by the time they got it all set up running, etc... I could have overnighted the data...

Depends on your situation I guess.

Good luck!

UPDATE:

You know, I got thinking about your problem a little more. Why does the file have to be a single huge tarball? Tar is perfectly capable of splitting large files into smaller ones (to span media for example) so why not split that huge tarball into more managable pieces and then transfer the pieces over instead?

SCP – How to Move Files with SCP?

rsync over ssh is probably your best bet with the --remove-source-files option

rsync -avz --remove-source-files -e ssh /this/dir remoteuser@remotehost:/remote/dir

a quick test gives;

[tomh@workstation001 ~]$ mkdir test1
[tomh@workstation001 ~]$ mkdir test2
[tomh@workstation001 ~]$ touch test1/testfile.1
[tomh@workstation001 ~]$ ls test1/
testfile.1
[tomh@workstation001 ~]$ rsync --remove-source-files -av -e ssh test1/testfile.1 tomh@localhost:/home/tomh/test2/
sending incremental file list

sent 58 bytes  received 12 bytes  10.77 bytes/sec
total size is 0  speedup is 0.00

[tomh@workstation001 ~]$ ls test1/
[tomh@workstation001 ~]$
[tomh@workstation001 ~]$ ls test2/
testfile.1

As @SvenW mentioned, -e ssh is the default so can be omitted.

Best Answer

Related Solutions

Linux – Copy large file from one Linux server to another

SCP – How to Move Files with SCP?

Related Topic