Centos – How to change boot device on vm installed on Citrix Xen Server to DVD

Well, seeing as though I pulled my hair out solving this, let me answer my question and save someone else the trouble of hair pulling :)

Solution:

After playing around, plenty of Googling and repartitioning etc... I came to a setup that works like a charm.

There is probably a quicker way to do this but I am not going to over complicate this answer

I did a standard install with the partitions like this (I have a 500g hard drive) :

/boot 100mb 
/swap 4gb 
/     40gb 

The balance of the disk space is to be left as unpartitioned space.

Then, I created a primary partition called /dev/sda4 by following these steps:

~: fdisk /dev/sda
~: (fdisk shell) p4   (for primary partition # 4)
~: (fdisk shell) t    (hit t and enter to edit the partition type)
~: (fdisk shell) 08e  (Linux LVM)

Reboot the server so that the new partitions can take effect.

Now create logical volumes by:

~: pvcreate /dev/sda4
~: vgcreate xenvg -s 4M /dev/sda4    # (xenvg is the name of my virtual group, you can rename it)
~: lvcreate -L400G -n xenroot xenvg  # (xenroot is going to be my drbd  resource and root partition for my DomU)
~: lvcreate -L4G  -n xenswap xenvg   # (xenswap is my swap file for my DomU)

Now that you have the correct partitioning you can go ahead and install DRBD with the following config file directives (drbd.conf)

Just displaying the 2 important directives here...

{
device  /dev/drbd0;
disk    /dev/xenvg/xenroot;
}

Your XEN VM config file needs to look like this (again, just the important one)

{
disk = [ "drbd:xenvm,xvda,w","phy:xenvg/xenswap,xvdb,w" ]
}

I hope this helps someone out there...

I don't suppose this for kernel modules or portage trees, is it? That's what I've seen this mechanism used for...

So, sure enough it's easy to have all of your guests have a filesystem image file attached to them as a read-only block device. It's also very straightforward to have that mounted somewhere in the guest (/etc/fstab and all that Jazz). Ownerships you'll presumably take care of in the block device anyway (assuming you're using a filesystem type that stores that metadata -- but if you're using, say, VFAT, ownership is only a mount option away anyway).

The trick is handling updates. Once you've got your "block device" mounted in any guest, nothing can be allowed to update it. It just won't work, because nobody knows that someone else is updating the contents, so everything falls apart. Instead, you need to create a copy of the file with the filesystem image, make whatever changes are needed, and then trigger some sort of update action to make the guests unmount the old "filesystem", then the dom0 can detach the old file and attach the new one, before the guest remounts the filesystem.

In the cases I've used this, we actually had some code in the domU config files (since they're just Python anyway) to find the newest of these block devices and attach that, then the usual boot-time mounts did the right thing. So, for us, the "update process" was "reboot the guest". Whether that works for you, though, is a question I can't answer because I don't know what you're trying to use this for.

Alternately, just have a second NFS server that is only used for supplying these files to your domUs. It's probably easier than all this block device frufru (we had some pretty specific requirements that made it the least-worst option, but I don't expect they apply in your case -- in fact, I know they don't apply in your case, because you've already got an NFS server).