Ssh – Backup remote VPS with Rsync

rsyncsshvps

I have a VPS that we will be shutting down soon, so I want to create a full backup of it. I was thinking about rsync, because if I google linux and backup words, I constantly get hit by rsync, so why the hell not. 🙂 However, here comes the kicker.

The backup machine where I want to backup the VPS in not accessible from the internet. So I can't just use what almost everybody suggests to SSH into the VPS, sudo rsync almost everything and as output mark the backup server. I want to run a bash command that does something similar to dd, so if I run rsync on the remote server, I could mark my local folder as output. Also I have to mention, that root ssh login is not allowed, so I can only SSH into the VPS as a user who can actually turn into sudo, but by default it is just a plain user.

I came up with the following rsync command:

sudo rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /* /backup/machine/folder

And I wish to use it somehow like this dd command I've used for cloning but the opposite way:

dd if=/dev/sda bs=4096 | pv | ssh root@<<TargetServer-IP>> ”dd of=/dev/sda bs=4096”

Best Answer

I love to post questions here because about 50% of them is being answered by myself. I usually take a few hours to decide to upload the question here, and after that a few mutes later, I get my answer. :) Here is the command:

rsync --dry-run root@omd.myserver.hu:/ -aAXvh --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /backup/backup/servers/omd

OFC exclude the --dry-run from the command if you want to execute it. Also, you have to add your SSH public key to /root/.ssh/authorized_keys file to make this work.

Related Solutions

RSYNC for Incremental backup takes 3+ days

I doubt that the upgrade alone will provide the sort of improvement you're looking for. At 72 hours, you'd probably want an order of magnitude performance increase (7.2 hours). If you're looking for 2-3 hours, good luck without an SSD and a good network.

With 55 million inodes (assuming approx as many files), you're going to have to seriously reconsider your approach. First, if you are using an ext variant I'd consider benchmarking a different FS.

Second, if you're using an ext FS (say ext3/4), the first thing I'd do is shut off atime! With atime on, every time a file is read/looked at the filesystem has to do a tiny write to the disk as atime means "access time". By shutting it off, you lose the ability to see when a file was accessed but that's the way the cookie crumbles. If you are using a standard SATA disk, assume you can do 100 IOs per second (IOPS). Each access write takes one of those (worst case). That means 100 files a second just to verify it's existence, and by the time you read it you are using even more IOPS. 55000000/100 = 550000s = 152 hours. Once you figure in the kernel's very good algorithms to merge the IOPS, you've probably found your bottleneck.

In your /etc/fstab, use the mount option:

noatime,nodiratime

to completely disable atimes. Leave off nodiratime to leave access times off for directories. If you have a lot of directories, I'd recommend turning it off.

I bet this alone will help dramatically.

Here's an example fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=66188c62-0d8c-46d8-a44f-8f673ca6ac98 /               ext4    errors=remount-ro,discard,noatime,nodiratime 0       1
# swap was on /dev/sda6 during installation
UUID=c3f40312-d6f9-4bb7-b426-602a4b7a6c47 none            swap    sw              0       0

Ssh – rsync as root to /var/www failing with “protocol version mismatch”

You can also start rsync using sudo on the remote machine to run it with root privileges. The option you can use for this is --rsync-path:

rsync -avL --progress -e "ssh -i /home/me/myhosts.pem" --rsync-path="sudo rsync" source-folder ubuntu@ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com:/var/www/

Related Topic