Would rsync be better than tar, wget and untar

migrationrsynctar

I am migrating to a new server and, as well as html/php files, have a directory which contains around 90,000 files totalling 24GB in size which needs to be moved.

When I did a test migration, I used tar to create a tarball then wget on the new host and then extracted the tar file but, whilst it worked fine, it took around 3 hours all in to complete. That would 3 hours of downtime whilst I did this on the actual migration to ensure no new files come in or files were changed etc.

I am now planning the actual migration and am trying to find quicker ways of doing this and wondered about using rsync – I have only ever used rsync locally and only for a small number of files so would running rsync against 90k items be quicker than the above method?

I am not so worried about CPU, memory or network usage as long as the actual process completes quicker as this will be done out of hours when the system is quieter anyway.

Best Answer

You will have to test to find out. There are many variables such as the speed of your storage system.

Consider restoring your tar archive or backup while the old system is up. Then during the downtime use rsync to copy the remaining changes. It still has to check many file modify times which takes time, but there is much less I/O and network transfer.

This incremental copy is what rsync is good at. Doing it in one go may not be faster than other tools.

bozosync.sh

Hmm, if you only want to move the changed files, why would the tar command be updating the archive? If only changed files are being sent, wouldn't the tar archive consist only of them, and so be necessarily recreated from scratch each time?

If I'm understanding this correctly you want something like this, I call it bozosync.sh:

TarFile=fun.tgz
TheScript=$0
result=
ls -1 . | (
  while read f; do
    if [ -f "$f" -a $f -nt $TheScript ]; then
      result="$result $f"
    fi
  done
  tar cvfz $TarFile $result
  touch $TheScript
)

This puts every plain file in the current directory in the tar archive, for each file that is newer than the script itself, and then it runs touch(1) on the script to update its mod time. This does have a race condition, in that a file that is modified in between the test and the touch will be ignored the next time around. If this is a problem you could get around that by copying the script prior to the loop, and then moving it back to itself at the end. (Then you still have a race but the failure is softer, it may simply include a file that hasn't really changed.)

Ubuntu – Rsync over ssh with root access on both sides

Actually you do NOT need to allow root athentication via SSH to run rsync as Antoine suggests. The transport and system authentication can be done entirely over user accounts as long as you can run rsync with sudo on both ends for reading and writing the files.

As a user on your destination server you can suck the data from your source server like this:

sudo rsync -aPe ssh --rsync-path='sudo rsync' boron:/home/fred /home/

The user you run as on both servers will need passwordless* sudo access to the rsync binary, but you do NOT need to enable ssh login as root anywhere. If the user you are using doesn't match on the other end, you can add user@boron: to specify a different remote user.

Good luck.

*or you will need to have entered the password manually inside the timeout window.

Best Answer

Related Solutions

Ssh & tar to incrementally transfer (like rsync)

bozosync.sh

Ubuntu – Rsync over ssh with root access on both sides

Related Topic