Linux – KVM/qemu – use LVM volumes directly without image file

kvm-virtualizationlinuxlvm

I'm not quite sure how to phrase this question (hence the poor title), so let me provide an example of what I'm trying to do.

On my (old) Xen host, I'm able to present LVM filesystems directly to each guest. These filesystems are actually created and formatted on the host, and passed directly through. Eg., for one of my hosts using a separate tmp and swap partitions, I define the storage like this:

disk = [

'phy:/dev/vg1/guest1-swap,sda1,w',

'phy:/dev/vg1/guest1-disk,sda2,w',

'phy:/dev/vg1/guest1-tmp,sda3,w',

]

So, guest1-swap is formatted as a swap partition, guest1-disk and guest1-tmp are formatted with ext4, and from the guest's perspective it simply sees them as three formatted partitions under /dev/sda.

(This may sound like a lot of work, but there are provisioning scripts, such as the awesome xen-tools, that automated pretty much everything).

This provides some really useful capabilities, two of which I'm especially interested in figuring out for KVM:

Mount the guest filesystems from the host OS. I can do a read-only mount of any guest filesystem at any time, even while the guest is running. This has the side benefit of allowing my to create LVM snapshots of any existing volume while the guest is running. This way, I'm able to centrally backup all my guests, while running, from the host.
Online volume resizing. Because the volumes contain standard Linux filesystems, I can use a combination of lvextend and resize2fs to grow my guest filesystems, again while they're online.

I'm currently setting up a KVM host that will replace the Xen host. Similar to the Xen setup I'm leveraging LVM to provide direct filesystem access, but KVM/qemu behaves differently in that it always creates a image file for the guests, even on the LVM volume. From the guest's perspective, it sees this as an unpartitioned disk, and it's up to the guest to apply a partition label, then create the partitions and filesystems.

From a guest perspective that's fine, but from a server/management perspective it seems to be far less flexible than the Xen setup I described. I'm still new to KVM, so I may be (hopefully) missing something.

I ran into this problem when trying to re-implement my former backup solution on the KVM host and the mount command chocked when I tried to mount one of the guest's filesystems. So, addressing that is my current concern, but it also made me concerned about the resizing thing, because I'm sure that issue will come up at some point as well.

So, here are my questions:

Is there any way to have kvm/qemu use LVM volume filesystems directly as I described for my Xen setup? I use libvirt for management if that makes a difference.
If not, what can I do to get similar mounting/backup functionality under KVM? I've seen discussions about using libguestfs w/ FUSE to do this, but is that really the best option? I'd prefer to stick with a native filesystem mount if at all possible.
Also if not, is it possible to do an online filesystem resize under KVM? I've found several discussions/howtos about this, but the answers seem to be all over the place with no clear, and definitely no straightforward, solutions.

Sorry for the long post, just wanted to make sure it was clear. Please let me know if I can provide any other info that would be helpful. Looking forward to the discussion. 🙂

Best Answer

qemu-kvm can use LVs as virtual disks instead of files. this is quite a common use case actually.
libguestfs (and just look for a set of virt-* tools) can provide access to guest filesystems in a cleaner way than anything you remount to the host directly, though both are possible.
Online FS resizing is not a feature of kvm, but something the guest OS should be capable of. resize2fs will work in a VM as well as it does on physical hardware, the only problem being the guest redetecting the size changes. Try virt-resize as the standard tool, but lvresize and qemu-img can also easily be used (though in offline mode, requiring a guest restart usually).

I think lvresize with resize2fs will actually work without a guest restart, but I haven't tried it yet

Related Solutions

Linux – Using LVM snapshots for KVM virtual machine clones

It's perfectly fine to do this. What you do not want is to have the parent of the snapshot (the original, or the source, or whatever you want to call it) to be in use at the same time, because it will cause IO multiplication (Hubert was right about this, it is just easy to prevent by not using the source volume all the time).

If you have one master OS install on an LVM, and you snapshot that four times, you will not have much of an IO penalty, since you are only writing to the individual snapshot volumes. Of course, it is not free, but neither are other forms of snapshotting on other filesystems or virtual disks. There is always a cost somewhere.

One other thing Hubert is right about, is that you have to think about the sizing of your snapshots. You will want to make sure the snapshot volumes are able to keep writing. A full snapshot volume will break stuff badly. A failsafe way to prevent this, is to make the snapshot volume the same size (or bigger) than the source volume. You loose the benefit of using less diskspace this way, though.

You know that qemu images are snapshot capable too?

Linux – Rescue disk is unable to see the lvm physical volumes

vgscan -vvvv

will give you very extensive output about why vgscan considers any specific volume being part of a volume group. You also could run pvs -a to see a summary of your physical volumes alongside with volume group assignments.

vgscan -vvvv output for one of the partitions:

Opened /dev/sdc3 RO /dev/sdc3: size is 3772817055 sectors 
Closed /dev/sdc3 /dev/sdc3: size is 3772817055 sectors 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
Closed /dev/sdc3 Using /dev/sdc3 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
/dev/sdc3: No label detected
Closed /dev/sdc3

pvs -a didn't reveal anything. All physical volumes listed without a volume group assignment

"no label detected" sounds pretty sad. You are sure that it is a LVM2 partition and not, say, a partition used by md-raid? You could check using mdadm --examine /dev/sdc3. And please post fdisk -l /dev/sdc

Yes I'm sure it is an LVM2 partition. The mdadm command gives "No md superblock detectd on /dev/sdc3" The fdisk says /dev/sdc3 is a Linux LVM partition.

Ah, then you will be in the lucky position (irony here, sorry) to try LVM recovery due to presumably damaged data structures. There is a howto about LVM recovery which might give you a starting point - try loading your VG configuration either from the disk itself (using dd if=/dev/sdc3 bs=512 count=255 skip=1) or from the /etc/lvm/backup folder of your former root filesystem (which I understand is on /dev/sdc1) into /etc/lvm/backup/ and re-issuing the vgscan command.

I tried that on both sda3 and sdc3 (as you can see, I have 3 lvm partitions to do this too) and they all result in binary files in the output text file. Ok, correction. There is some lvm meta data in the file, but it's several bytes into the file. I'm looking through the data, but it looks correct. I will keep trying to go through that restore process.

This is expected - both, VG and LV configs is cleartext within binary structures.

I ended up using a slightly modified process than what was outlined here. I ended up making a cfgbackup file from the byte data in the LVM, do a pvcreate, then a vgcfgrestore. After that, it worked. Thanks for the help.

Best Answer

Related Solutions

Linux – Using LVM snapshots for KVM virtual machine clones

Linux – Rescue disk is unable to see the lvm physical volumes

Related Topic