Handling XenServer snapshotting and cloning with ZFS SAN

nexentasnapshotvdixenserverzfs

I'm planning on setting up a XenServer machine that uses a NexentaOS/ZFS-based SAN for storage of the Virtual Disk Images (VDIs) through iSCSI. I know I could just setup a big Storage Repository (SR) on the SAN and let XenServer take care of snapshotting and cloning disk images. However, I'd love to tap more into the power of ZFS and use that for snapshotting/cloning, for a few reasons:

  1. I'm not sure how XenServer's snapshotting/cloning works, but if it's based on LVM, I'm concerned I'd run into issues when dealing with multiple snapshots. I've done some experiments a while ago with multiple LVM snapshots of the same data and performance seemed poor and the snapshots wasted a lot of space. It seems that ZFS snapshots are far superior to LVM snapshots.
  2. The SAN would be taking automatic (and efficient) periodic ZFS snapshots that could go back in time a while, and I'd love to be able to revert a VM to such ZFS snapshot.

Would letting ZFS handle snapshotting/cloning instead of doing it through XenServer be advisable, and if so, what's the best way to go about it? If I put all VDIs inside a single large SR, and take ZFS snapshots of the entire SR, I would not be able to roll back an individual VM at a time. I could create one SR per VDI, and if I have to rollback a VDI, I'd carefully detach the SR, roll it back on the SAN, and re-attach it. However, I'm guessing that I'd run into problems when attaching a cloned SR if XenServer detects duplicate SR UUIDs. Are there any better ways to handle cloning or rolling back to previous snapshots from the SAN?

Best Answer

As other answers alluded to, the ideal approach is LUN-per-VDI. At first it didn't look like it was possible to do this, but there is an undocumented "iscsi" SR driver that will create a LUN-per-VDI SR (I found this when looking through the /opt/xensource/sm directory - see the ISCSISR.py file). You essentially setup one SR for each iSCSI target, and XenServer creates the VDIs for each LUN on that target. You can only set this up through the command-line, including creating the VBDs and attaching it to VMs. The VBDs and VDIs don't even show up in XenCenter.

Here's a sample command to set it up:

xe sr-create name-label=san type=iscsi device-config:target=<IP address of target> device-config:targetIQN=<IQN of target> shared=true

This will automatically create an VDI for each iSCSI LUN. If you end up adding a new iSCSI LUN on the SAN, XenServer will add a new VDI for it after executing the following command:

xe sr-scan uuid=<UUID of SR>

This also works when adding VDIs for cloned LUNs - the new VDI gets assigned a new UUID.

Also, if you end up resizing a LUN, XenServer does not automatically pick up on that, so you'd have to execute the following:

xe vdi-forget uuid=<UUID of VDI that you resized>
iscsiadm -m node -R
xe sr-scan uuid=<UUID of SR>

And to create a VBD and attach it to a VM:

xe vbd-create vm-uuid=<UUID of VM> device=<VBD device #, e.g. 0> vdi-uuid=<VDI of UUID>
Related Topic