Ftp – How to do automated FTP backups – and how to restore them

automationbackupftpvirtualization

I'm currently using a Virtual Server running Ubuntu Linux. The company hosting it does not do any kind of backups, but they offer me the oppertunity to do backups via FTP. Therefor, I get access to a FTP share of equal size to my primary server storage. There is no more help availbale than This is your FTP account data, have fun! but I guess I'm supposed to write a small script and run it as cron job. They promote their backup solution as if it's the best and easiest thing in the world, but right now I see many open questions.

So I'm not asking on how to write a shell script, but rather on what it should do.

First, I'll have to decide on whether I just want to backup some data folders (like the htdocs and a recent DB dump) or if I want to mirror the complete server. I'd like to do the latter, because I'm not only concerned about data loss, but going through the pains of server configuration from scratch, which usually keeps my busy for some days.
Next, I wonder if I will get any problems by backing up files which are currently accessed by some process, maybe even written to. Will I end up with corrupted files in my backup, will the backup just stop, or will the normal operation of the server be interupted?
If I had twice the backup space as my primary server, then I would upload the backup n, and after success, delte the backup n-1. But I only have space equal to my server's hard disk, so I cannot keep backup n and n-1 at the same time. But if I do it the other way round (that is, delete backup n-1 and then start uploading backup n) I'm going through times without any backup at all, which I'd like to avoid.
If I had physical access to that server, or at least access to the file cointing the virtual hard disk image of my VServer, I'd just backup that image and restore it as a whole. But sadly, I don't have that access.
And last thing, I'm also wondering how I'm supposed to restore that data in case the primary storage goes away. Is it possible, on Linux, to let the system overwrite itself with the backup data while it's running? Even if it is possible to tell the system to download everything via ftp, store it right at the file system root / and overwrite everything what's there, I'd still have to delete all the files which are there now but were not present in the backup.

Any help and ideas are much appreciated!

Best Answer

Try ftplicity - a nice howto can be found at HowToForge: http://www.howtoforge.com/ftp-backups-with-duplicity-ftplicity-debian-etch

To answer your other questions:

it makes sense to only backup the server configuration (/etc), your user-data (/home, /usr/local, ...) and not the whole server. A restore of ubuntu can easily be done with dpkg, see "dpkg get-selections" at ubuntugeek.com/clone-your-ubuntu-installation.html
yes, files which are currently being written will give you backup problems (MySQL databases, Mail spoolfiles etc) so it's usually best practice to have a look at all the services that you need backuped and look into their documentation
with not backing up the whole server and the help of tar you should be able to store multiple backup sets at the ftp location
restoration should be simple: you'll get a working virtual server, right? Use dpkg set-selections to upgrade the packages, restore /etc, reboot (or reload the services) and copy your user-date back

Related Solutions

Linux – Help on writing Linux script to backup and ftp files to another server

I would not recommend using FTP for any files you care about (particularly database backups if that database contains customer information), but that is ultimately up to you.

To start with, script the creation of your backups on the Linux server with something like the following:

#!/bin/bash
# 
# Call script with:
# /cron-mysql_bkup.sh mysql_hostname mysql_username mysql_password /backup/directory
# 
if [ -n "$1" ]; then
 MYSQL_HOST="$1"
else
 echo "MYSQL_HOST not specified."
 exit 1
fi
if [ -n "$2" ]; then
 MYSQL_USER="$2"
else
 echo "MYSQL_USER not specified."
 exit 1
fi
if [ -n "$3" ]; then
 MYSQL_PASS="$3"
else
 echo "MYSQL_PASS not specified."
 exit 1
fi
if [ -n "$4" ]; then
 DIR_BKUP="$4"
 mkdir -p ${DIR_BKUP}
else
 echo "DIR_BKUP not specified."
 exit 1
fi
# 
# Generate filename
# 
BACKUP_FILENAME=`date "+%Y-%m-%d"`".sql"
# 
# Dump all databases
# 
mysqldump -h${MYSQL_HOST} -u${MYSQL_USER} -p${MYSQL_PASS} --all-databases > ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME}
# 
# Write status of the operation to the mysql.log file in the backup directory
# 
if [ $? -ne 0 ]; then
 echo `date`" - ERROR creating backups of host "${MYSQL_HOST} >> ${DIR_BKUP}/mysql.log
else
 echo `date`" - MySQL Backup complete for "${MYSQL_HOST} >> ${DIR_BKUP}/mysql.log
 BACKUP_TAR_NAME=${BACKUP_FILENAME}".tar.gz"
 tar czf ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_TAR_NAME} ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME} && rm ${DIR_BKUP}/${MYSQL_HOST}/${BACKUP_FILENAME}
fi

This script will:

Check to see whether all the required parameters were set
Dump all MySQL databases to a file named YYYY-mm-dd.sql under the backup directory specified (this directory will be created if it does not exist)
GZIP the file and, if the GZIP operation succeeds, delete the original SQL file

This script should be fairly straightforward for you to customize if you wish to - see the Advanced Bash Scripting Guide to learn more about doing so.

... once your databases are backed up and can be found in a specific location, why not automate backup retrieval with WinSCP instead of transferring your files over FTP?

Docker – How to Backup and Restore Docker Volumes

You're right. Since you can have multiple containers with volumes on their own, you need to keep track which volume corresponds to which container. How to do that depends on your setup: I use the name -data for the data container, so it's obvious to which container a image belongs. That way it can be backed up like this:

VOLUME=`docker inspect $NAME-data | jq '.[0].Volumes["/path/in/container"]'`
tar -C $VOLUME . -czvf $NAME.tar.gz

Now you just need to rebuild your image and recreate your data container:

cat $NAME.tar.gz | docker run -name $NAME-data -v /path/in/container \
                              -i busybox tar -C /path/int/container -xzf -

So this means you need to backup:

Dockerfile
volume
volume path in container
name of the container the volume belongs to

Update: In the meanwhile I created a tool to backup containers and their volume(s) (container(s)): https://github.com/discordianfish/docker-backup and a backup image that can create backups and push them to s3: https://github.com/discordianfish/docker-lloyd

Best Answer

Related Solutions

Linux – Help on writing Linux script to backup and ftp files to another server

Docker – How to Backup and Restore Docker Volumes

Related Topic