Backup to Remote Server and Create Tar on the Fly – Linux

linuxrsynctar

I would like to backup a directory from my production server to a backup server using rsync.

My current solution:

tar directory
on production server send to remote server via rsync

My Problem with this solution: there is not enough space to create the tar file + it needs to much resources while the tar is created. It should make as less as possible impact on the production server resources.

I would like to send the files into a tar file on the remote server. I simply could send the files and tar them on the backup server, but in my point of view, this would not be a good solution.

Best Answer

You could use a combination of pipes and STDOUT to send the tar file over the network to the remote server.

tar zcvf - /your/directory | ssh -i private.key backup-user@backupserver "cat > /backup/file.tgz"

The "-" character sends the tar to STDOUT which is piped to ssh and the contents written to a regular file using cat.

You'll need to use ssh with a private key file for "passwordless" authentication.

This doesn't use rsync but will get the tarfile to the remote server.

Reference: https://stackoverflow.com/questions/8045479/whats-the-magic-of-a-dash-in-command-line-parameters

bozosync.sh

Hmm, if you only want to move the changed files, why would the tar command be updating the archive? If only changed files are being sent, wouldn't the tar archive consist only of them, and so be necessarily recreated from scratch each time?

If I'm understanding this correctly you want something like this, I call it bozosync.sh:

TarFile=fun.tgz
TheScript=$0
result=
ls -1 . | (
  while read f; do
    if [ -f "$f" -a $f -nt $TheScript ]; then
      result="$result $f"
    fi
  done
  tar cvfz $TarFile $result
  touch $TheScript
)

This puts every plain file in the current directory in the tar archive, for each file that is newer than the script itself, and then it runs touch(1) on the script to update its mod time. This does have a race condition, in that a file that is modified in between the test and the touch will be ignored the next time around. If this is a problem you could get around that by copying the script prior to the loop, and then moving it back to itself at the end. (Then you still have a race but the failure is softer, it may simply include a file that hasn't really changed.)

Concurrent modification during backup: rsync vs dump vs tar vs

None of the tools you are considering provide guarantees against concurrent modification. However, do you really need a point in time snapshot. If so use the LVM snapshot option given above. As you have given rsync as an option I assume that disk to disk backup is an option.

Least secure is dump which takes a copy of the disk blocks as they are read. Given the size of your data there is likely to be significant differences between the directory information and the data. For disk to disk backup you could consider dd to partitions of the same size as an alternative. Both solutions do essentially the same thing and have the same problems.

Tar will read the files one by one and will read to the end of each. If a file is renamed or deleted while tar is backing it up tar will back up the file it started reading. It is a reasonable solution for log files.

Rsync behaves like tar, but only copies changes. Essentially it will copy all changes to the directories. With a date based log rotation scheme (logfile.yyymmdd) instead of the common rotated version scheme (logfile.1 logfile.2gz ...). It can efficiently backup your logfiles.

Best Answer

Related Solutions

Ssh & tar to incrementally transfer (like rsync)

bozosync.sh

Concurrent modification during backup: rsync vs dump vs tar vs

Related Topic