Linux device-mapper maps LVM PV nested inside LV when taking snapshot

device-mapperlinuxlvm

Which is really messing with my plan to back up this machine…

I have a server which is a KVM hypervisor to several virtual machines. One of these is running Docker. It has its Docker volumes on /dev/vdb, which is set up as an LVM PV, on which Docker uses its direct-lvm driver to store Docker container data. This virtual disk is an LVM LV on the host's local disk.

Both host and guest run Fedora 21.

The host's view of this volume is (only the relevant volume is shown):

[root@host ~]# lvs
  LV                           VG         Attr       LSize
  docker2.example.com-volumes vm-volumes -wi-ao---- 40.00g
[root@host ~]# dmsetup ls --tree
vm--volumes-docker2.example.com--volumes (253:10)
 └─ (9:125)

The guest's view of this volume is (again, only the relevant volume is shown):

[root@docker2 ~]# pvs
  PV         VG             Fmt  Attr PSize  PFree
  /dev/vdb   docker-volumes lvm2 a--  40.00g    0

With all the other LVM volumes on the host, I can take a snapshot with lvcreate --snapshot, backup the snapshot and then lvremove it with no issue. But with this particular volume, I can't lvremove it because it is in use:

[root@host ~]# lvremove /dev/vm-volumes/snap-docker2.example.com-volumes 
  Logical volume vm-volumes/snap-docker2.example.com-volumes is used by another device.

Eventually I figured out that device-mapper on the host had somehow figured out that this logical volume snapshot contained an LVM PV, and then proceeded to map the logical volumes within the snapshot to the host (only the relevant volumes are shown):

[root@host ~]# dmsetup ls --tree
vm--volumes-docker2.example.com--volumes (253:10)
 └─vm--volumes-docker2.example.com--volumes-real (253:14)
    └─ (9:125)
docker--volumes-docker--data (253:18)
 └─vm--volumes-snap--docker2.example.com--volumes (253:16)
    ├─vm--volumes-snap--docker2.example.com--volumes-cow (253:15)
    │  └─ (9:125)
    └─vm--volumes-docker2.example.com--volumes-real (253:14)
       └─ (9:125)
docker--volumes-docker--meta (253:17)
 └─vm--volumes-snap--docker2.example.com--volumes (253:16)
    ├─vm--volumes-snap--docker2.example.com--volumes-cow (253:15)
    │  └─ (9:125)
    └─vm--volumes-docker2.example.com--volumes-real (253:14)
       └─ (9:125)

These correspond exactly to the logical volumes inside the VM:

[root@docker2 ~]# lvs
  LV          VG             Attr       LSize
  docker-data docker-volumes -wi-ao---- 39.95g
  docker-meta docker-volumes -wi-ao---- 44.00m

Notably, it doesn't try to do this to the LVM LV when the system is booting, but only when I take a snapshot.

What is going on here? I really don't want device-mapper inspecting the contents of LVM snapshots to see if there's anything within them it can unhelpfully map for me. Can I suppress this behavior? Or do I need to create the snapshot via some other method?

Best Answer

Sometimes the relevant documentation is hidden away in configuration files rather than in, say, the documentation. So it seems with LVM.

By default LVM will automatically attempt to activate volumes on any physical devices which get connected to the system after boot, so long as all of the PVs are present, and lvmetad and udev (or more recently systemd) are running. When the LVM snapshot gets created, a udev event gets fired off, and since the snapshot contains a PV, lvmetad automatically runs pvscan, and so forth.

By looking at /etc/lvm/backup/docker-volumes I was able to determine that lvmetad had explicitly run pvscan on the snapshot by using the device major and minor numbers, which bypassed LVM filters that would normally prevent this. The file contained:

description = "Created *after* executing 'pvscan --cache --activate ay 253:13'"

This behavior can be controlled by setting the auto_activation_volume_list in /etc/lvm/lvm.conf. It allows you to set which volume groups, volumes, or tags are allowed to be activated automatically.

So, I simply set the filter to contain both of the volume groups for the host; anything else won't match the filter and does not get automatically activated.

auto_activation_volume_list = [ "mandragora", "vm-volumes" ]

The guest's LVM volumes are no longer appearing on the host, and finally, my backups are running...

Related Topic