Transfer a 30GB tar file from one remote server to another – restrained by disk space

filestarwget

I have a 38GB "Moodle" tar file that I need to transfer from the old server (Shared hosting), which we can call "Server A" to the new server (Dedicated virtual), or "Server B". I'm with Mediatemple if that helps.

I've already tarballed the directory and used wget to transfer it to server B via SSH, however – it's a puny 100GB package and I'm at 96% disk capacity – meaning I cannot untar the file on server B! Is there any way I can transfer this huge file from Server A to Server B, preserving permissions and with least possible chance file corruption, which will not make me hit the disk limit?

While I have spent quite a while doing this so far, I would be willing to abandon my original plan if someone can offer a better idea – and I would be very grateful!

Best Answer

Don't tar. Use rsync -av to preserve permissions while transfering the files. Though like tar, this does not preserve selinux context. Not that I would consider that important though.

Related Solutions

Linux – Copy large file from one Linux server to another

Sneakernet Anyone?

Assuming this is a one time copy, I don't suppose its possible to just copy the file to a CD (or other media) and overnight it to the destination is there?

That might actually be your fastest option as a file transfer of that size, over that connection, might not copy correctly... in which case you get to start all over again.

rsync

My second choice/attempt would be rsync as it detects failed transfers, partial transfers, etc. and can pick up from where it left off.

rsync --progress file1 file2 user@remotemachine:/destination/directory

The --progress flag will give you some feedback instead of just sitting there and leaving you to second guess yourself. :-)

Vuze (bittorrent)

Third choice would probably be to try and use Vuze as a torrent server and then have your remote location use a standard bitorrent client to download it. I know of others who have done this but you know... by the time they got it all set up running, etc... I could have overnighted the data...

Depends on your situation I guess.

Good luck!

UPDATE:

You know, I got thinking about your problem a little more. Why does the file have to be a single huge tarball? Tar is perfectly capable of splitting large files into smaller ones (to span media for example) so why not split that huge tarball into more managable pieces and then transfer the pieces over instead?

Linux – “Stream” tar action from remote server through Putty

If you have cygwin, then you can just run something like this on your local machine:

ssh user@host 'tar cj directory' > /cygdrive/c/archive.tar.bz2

and it will put the tar archive in c:\archive.tar.bz2

If you don't have cygwin, you can do the same thing with plink (part of PuTTY) from a command window;

plink user@host 'tar cj directory' > c:\archive.tar.bz2

(I'm assuming you're on Windows if you're using PuTTY - if you're on Linux, just run the ssh command)

Best Answer

Related Solutions

Linux – Copy large file from one Linux server to another

Linux – “Stream” tar action from remote server through Putty

Related Topic