Linux – Backup Xen domU machines while running

LVM snapshots are meant to capture the filesystem in a frozen state. They are not meant to be a backup in and of themselves. They are, however, useful for obtaining backup images that are consistent because the frozen image cannot and will not change during the backup process. So while you won't use them directly to make long-term backups, they will be of great value in any backup process that you decide to use.

There are a few steps to implement a snapshot. The first is that a new logical volume has to be allocated. The purpose of this volume is to provide an area where deltas (changes) to the filesystem are recorded. This allows the original volume to continue on without disrupting any existing read/write access. The downside to this is that the snapshot area is of a finite size, which means on a system with busy writes, it can fill up rather quickly. For volumes that have significant write activity, you will want to increase the size of your snapshot to allow enough space for all changes to be recorded. If your snapshot overflows (fills up) both the snapshot will halt and be marked as unusable. Should this happen, you will want to release your snapshot so you can get the original volume back online. Once the release is complete, you'll be able to remount the volume as read/write and make the filesystem on it available.

The second thing that happens is that LVM now "swaps" the true purposes of the volumes in question. You would think that the newly allocated snapshot would be the place to look for any changes to the filesystem, after all, it's where all the writes are going to, right? No, it's the other way around. Filesystems are mounted to LVM volume names, so swapping out the name from underneath the rest of the system would be a no-no (because the snapshot uses a different name). So the solution here is simple: When you access the original volume name, it will continue to refer to the live (read/write) version of the volume you did the snapshot of. The snapshot volume you create will refer to the frozen (read-only) version of the volume you intend to back up. A little confusing at first, but it will make sense.

All of this happens in less than 2 seconds. The rest of the system doesn't even notice. Unless, of course, you don't release the snapshot before it overflows...

At some point you will want to release your snapshot to reclaim the space it occupies. Once the release is complete, the snapshot volume is released back into the volume, and the original remains.

I do not recommend pursuing this as a long-term backup strategy. You are still hosting data on the same physical drive that can fail, and recovery of your filesystem from a drive that has failed is no backup at all.

So, in a nutshell:

• Snapshots are good for assisting backups
• Snapshots are not, in and of themselves, a form of backup
• Snapshots do not last forever
• A full snapshot is not a good thing
• Snapshots need to be released at some point
• LVM is your friend, if you use it wisely.

There's a couple of potential pitfalls here. First of all, you mention dd for backups. While dd can be a wonderful tool, it's not an ideal backup tool, and if you're referring to using it to backup a live machine, it's a very poor choice. You're going to have some really badly inconsistent data and may not end up with a working image file.

Also, one note for the future. . . always setup your disk layout with LVM (and leave a few GB unallocated in the Volume Group), as it makes these kinds of situations about a hundred times easier to deal with. If the partitions were logical volumes under LVM, you could snapshot them, and then make use of the snapshot to give you a fairly consistent image.

Now, how critical is it that this box stay up? If you can take it down for a few minutes, there are P2V (Physical to Virtual) converters that you can use to make this transition a lot easier.

What kind of hardware configuration are you dealing with? If you have mirrored disks (RAID1) you might be able to break the mirror, pull a disk, and bring up the yanked disk in a second box. You could then clone that non-production copy.

Another important thing to consider, is what are you using for Xen? Is it RHEL and the Xen that's included? Or is it Citrix XenServer (now free)? This makes a difference as they both expect (by default) a slightly different disk format. Manually building an image for XenServer is a bit more complicated than building one for stock Xen.

It may sound a little unorthodox, but I've actually had better success in this type of situation by doing a simple base Linux install on a separate box, then running rsync from the production box to the new box, and then converting the non-production rsync'ed clone into the Xen domU (I've even had some luck just rsynch'ing from the production box to a Xen domU). Just run rsync (I recommend the options -HavSux) the old box to the new one while it's running, and once the rsync has completed, run rsync again. The first one will take a while to run, and will leave you with a lot of inconsistencies. The second one will run faster and because it only has to pull the changed stuff this time, put things in a more consistent state.

Whether that will be "good enough" for you, with regards to consistency, will depend on your requirements and what applications you're running. For example, if you've got an active database server on the production box, you'd definitely not want to leave rsync'ed file backups as "good enough". Instead, after the second rsync you'd take a database backup using the database tools, and import that on the clone. Or you'd wait to do that final synchronization until you have your xen domU up and running.

Good luck, and plan on a couple of dry runs and a couple more practice runs to get things working smoothly before you go for real.