Lvm – rsnapshot LVM fails to remove snapshot volume

lvmrsnapshot

I am using rsnapshot on a Debian Wheezy server. This was recently upgraded from squeeze. Since the upgrade, I am getting the following error from the hourly cron job:

remote rm -rf /share/HDA_DATA/backup/rsnapshot/hourly.3 p1=-rf p2=/backup/rsnapshot/hourly.3/
remote cp -al /share/HDA_DATA/backup/rsnapshot/hourly.0 /share/HDA_DATA/backup/rsnapshot/hourly.1 p1=-al p2=/backup/rsnapshot/hourly.0
  Logical volume "rsnapshot" successfully removed
  Logical volume "rsnapshot" successfully removed
  Unable to deactivate open raid5-dl-real (254:4)
  Failed to resume dl.
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot hourly 
----------------------------------------------------------------------------
ERROR: Removal of LVM snapshot failed: 1280

Two LVM volumes are correctly backed up and Logical volume "rsnapshot" successfully removed, but then it gets to volume dl in lvm VG raid5 and see the unable to deactivate raid5-dl-real.

The name of my lvm snapshot is called raid5/rsnapshot. raid5-dl-real does not correspond to a volume name – the real device is /dev/mapper/raid5-dl.

So if this is the dl volume itself, why would lvm be trying to deactivate it?

Note that this was happening to an entirely different volume originally, so I removed it from the backup. Now it has shifted to this one.

The rsnapshot log isn't very enlightening either:

[16/Jul/2013:17:26:26] /sbin/lvcreate --snapshot --size 512M --name rsnapshot /dev/raid5/dl
[16/Jul/2013:17:26:29] /bin/mount /dev/raid5/rsnapshot /mnt/lvm-snapshot
[16/Jul/2013:17:26:32] chdir(/mnt/lvm-snapshot)
[16/Jul/2013:17:26:32] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded . /backup/rsnapshot/hourly.0/dl/
[16/Jul/2013:17:27:57] rsync succeeded
[16/Jul/2013:17:27:57] chdir(/root)
[16/Jul/2013:17:27:57] /bin/umount /mnt/lvm-snapshot
[16/Jul/2013:17:27:58] /home/share/scripts/rsnapshot_lvremove --force /dev/raid5/rsnapshot
[16/Jul/2013:17:29:02] /usr/bin/rsnapshot hourly: ERROR: Removal of LVM snapshot failed: 1280
[16/Jul/2013:17:29:02] rm -f /var/run/rsnapshot.pid

Any ideas?

Update – this has just started happening on an entirely different server. Same LVM issue.

One thing I have tried is to redirect the lvremove command to a script:

#!/bin/bash
sync
sleep 600
ls /dev/mapper/raid5-*-real
for i in /dev/mapper/raid5-*-real; do /sbin/dmsetup remove $i ; done
/sbin/lvremove --debug "$@"

So this syncs, sleeps for a bit, then removes any -real device maps before attempting the lvremove.

Even after all of this, the removal often fails. Here is the output from rsnapshot. Please ignore the error part way through, while there is an issue on one of the volumes, it is not until later that the lvremove fails:

remote cp -al /share/HDA_DATA/backup/rsnapshot/hourly.0 /share/HDA_DATA/backup/rsnapshot/hourly.1 p1=-al p2=/backup/rsnapshot/hourly.0
  One or more specified logical volume(s) not found.
/dev/mapper/raid5-crypt-real
/dev/mapper/raid5-db-real
device-mapper: remove ioctl on raid5-crypt-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db-real failed: Device or resource busy
Command failed
  Logical volume "rsnapshot" successfully removed
  One or more specified logical volume(s) not found.
/dev/mapper/raid5-crypt-real
/dev/mapper/raid5-db-real
/dev/mapper/raid5-db--var-real
device-mapper: remove ioctl on raid5-crypt-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db--var-real failed: Device or resource busy
Command failed
  Logical volume "rsnapshot" successfully removed
  One or more specified logical volume(s) not found.
/dev/mapper/raid5-crypt-real
/dev/mapper/raid5-db-real
/dev/mapper/raid5-db--var-real
device-mapper: remove ioctl on raid5-crypt-real failed: Device or resource busy
Command failed
device-mapper: remove ioctl on raid5-db-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db--var-real failed: No such device or address
Command failed
  /dev/raid5/rsnapshot: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/raid5/rsnapshot: read failed after 0 of 4096 at 42949664768: Input/output error
  /dev/raid5/rsnapshot: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5/rsnapshot: read failed after 0 of 4096 at 4096: Input/output error
  Logical volume "rsnapshot" successfully removed
  One or more specified logical volume(s) not found.
/dev/mapper/raid5-crypt-real
/dev/mapper/raid5-db-real
/dev/mapper/raid5-db--var-real
/dev/mapper/raid5-dl-real
device-mapper: remove ioctl on raid5-crypt-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-db--var-real failed: No such device or address
Command failed
device-mapper: remove ioctl on raid5-dl-real failed: Device or resource busy
Command failed
  Unable to deactivate open raid5-dl-real (254:25)
  Failed to resume dl.
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot hourly 
----------------------------------------------------------------------------
ERROR: Removal of LVM snapshot failed: 1280

Best Answer

In the case this can help anyone, I had the problem as described in the Debian bug id 659762 report.

I identified a volume in suspended state with dmsetup info and reactivated it with dmsetup resume This unlocked the LVM system.

Related Solutions

LVM snapshots as a backup strategy

LVM snapshots are meant to capture the filesystem in a frozen state. They are not meant to be a backup in and of themselves. They are, however, useful for obtaining backup images that are consistent because the frozen image cannot and will not change during the backup process. So while you won't use them directly to make long-term backups, they will be of great value in any backup process that you decide to use.

There are a few steps to implement a snapshot. The first is that a new logical volume has to be allocated. The purpose of this volume is to provide an area where deltas (changes) to the filesystem are recorded. This allows the original volume to continue on without disrupting any existing read/write access. The downside to this is that the snapshot area is of a finite size, which means on a system with busy writes, it can fill up rather quickly. For volumes that have significant write activity, you will want to increase the size of your snapshot to allow enough space for all changes to be recorded. If your snapshot overflows (fills up) both the snapshot will halt and be marked as unusable. Should this happen, you will want to release your snapshot so you can get the original volume back online. Once the release is complete, you'll be able to remount the volume as read/write and make the filesystem on it available.

The second thing that happens is that LVM now "swaps" the true purposes of the volumes in question. You would think that the newly allocated snapshot would be the place to look for any changes to the filesystem, after all, it's where all the writes are going to, right? No, it's the other way around. Filesystems are mounted to LVM volume names, so swapping out the name from underneath the rest of the system would be a no-no (because the snapshot uses a different name). So the solution here is simple: When you access the original volume name, it will continue to refer to the live (read/write) version of the volume you did the snapshot of. The snapshot volume you create will refer to the frozen (read-only) version of the volume you intend to back up. A little confusing at first, but it will make sense.

All of this happens in less than 2 seconds. The rest of the system doesn't even notice. Unless, of course, you don't release the snapshot before it overflows...

At some point you will want to release your snapshot to reclaim the space it occupies. Once the release is complete, the snapshot volume is released back into the volume, and the original remains.

I do not recommend pursuing this as a long-term backup strategy. You are still hosting data on the same physical drive that can fail, and recovery of your filesystem from a drive that has failed is no backup at all.

So, in a nutshell:

Snapshots are good for assisting backups
Snapshots are not, in and of themselves, a form of backup
Snapshots do not last forever
A full snapshot is not a good thing
Snapshots need to be released at some point
LVM is your friend, if you use it wisely.

Linux – Any way to recover ext4 filesystems from a deleted LVM logical volume

Every time you perform an operation with LVM, by default, the previous metadata is archived in /etc/lvm/archive. You can use vgcfgrestore to restore it, or grab the extends by hand (harder, but lvcreate(8) should cover it).

Edit:

And to make it as easy as possible, I should add that you can find the last backup before your destructive operation by looking at descriptions:

# grep description /etc/lvm/archive/vg01_*
/etc/lvm/archive/vg01_00001.vg:description = "Created before executing 'lvremove -f /dev/vg01/foo'"
/etc/lvm/archive/vg01_00002.vg:description = "Created before executing 'lvremove -f /dev/vg01/bar'"
/etc/lvm/archive/vg01_00003.vg:description = "Created before executing 'lvremove -f /dev/vg01/baz'"

Edit:

The normal allocation policy (default one) will allocate a stripe from the first free PE when there is enough room to do so. If you want to confirm where the LV was allocated, you can look in the archive files, those are perfectly readable by humans.

Best Answer

Related Solutions

LVM snapshots as a backup strategy

Linux – Any way to recover ext4 filesystems from a deleted LVM logical volume

Related Topic