Lvm – User reduced LVM logical volume without resizing filesystem

ext3lvm

I received an email yesterday that one of our users was trying to make room for a heartbeat/clustering package which requires its own partition to act as a voting disk. To do this, he attempted to reduce the size of the root partition's logical volume, and then create a new logical volume for this purpose. However, he forgot to resize the filesystem first (or include the -r switch in the command). He also forgot to unmount the root partition by running this process from a rescue cd. The system is now refusing to boot into the OS with the following error:

Either the superblock or the partition table is likely to be corrupt!
Unexpected Inconsistency; run fsck manually.

The system them drops the user into single user mode.

Is it possible to rescue the filesystem, or is it hosed? Its running ext3.

Best Answer

If a filesystem has already been initialised on the new LV then you're probably hosed, because data will have been overwritten. You can still try to recover it, but I wouldn't be too hopeful. If a filesystem hasn't been initialised then it's theoretically possible to recover it (but I haven't tried this myself).

Firstly, make a backup of the entire drive, you'll need it in case the recovery goes wrong. The next step is to try find the LVM metadata backups which LVM creates in /etc/lvm/archive before making changes. If the root volume isn't accessible you can try running e2fsck to get it mountable, and hope that the backup files were stored near the beginning of the volume. If you do this you'll want to continue recovery from before you ran e2fsck (i.e. restore from the backup after getting the backup files).

If you managed to get a backup file, restore the LVM configuration with vgcfgrestore. If you couldn't get a backup file you're just going to have to hope that the initial volume was completely sequential. Remove the new LV and then extend the old LV to its original size.

Once you've got the old LV back to the correct size, cross your fingers and run e2fsck. And you really need to make that backup first, you're probably not going to get this right on the first attempt.