Linux – Rescue disk is unable to see the lvm physical volumes

linuxlvm

We have a system that is setup to have a redundant filesystem including the boot partitions. We are at a point now that the system does not boot properly on either boot partitions. When booting, it says it can't locate the volume group.

When booting on the CentOS rescue disk, fdisk -l sees all the partitions:

1st Raid has
/dev/sda1 Boot
/dev/sda2 swap
/dev/sda3 LVM

2nd Raid has
/dev/sdc1 Boot
/dev/sdc2 swap
/dev/sdc3 LVM

There is also a 3rd LVM on a raid disk that is there for only data. This is not being read either.

When running pvscan, it says "No matching physical volumes found". Is there a way to see why it does not recognize the LVM physical volumes?

Best Answer

vgscan -vvvv

will give you very extensive output about why vgscan considers any specific volume being part of a volume group. You also could run pvs -a to see a summary of your physical volumes alongside with volume group assignments.

vgscan -vvvv output for one of the partitions: 

Opened /dev/sdc3 RO /dev/sdc3: size is 3772817055 sectors 
Closed /dev/sdc3 /dev/sdc3: size is 3772817055 sectors 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
Closed /dev/sdc3 Using /dev/sdc3 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
/dev/sdc3: No label detected
Closed /dev/sdc3

pvs -a didn't reveal anything. All physical volumes listed without a volume group assignment

"no label detected" sounds pretty sad. You are sure that it is a LVM2 partition and not, say, a partition used by md-raid? You could check using mdadm --examine /dev/sdc3. And please post fdisk -l /dev/sdc

Yes I'm sure it is an LVM2 partition. The mdadm command gives "No md superblock detectd on /dev/sdc3" The fdisk says /dev/sdc3 is a Linux LVM partition.

Ah, then you will be in the lucky position (irony here, sorry) to try LVM recovery due to presumably damaged data structures. There is a howto about LVM recovery which might give you a starting point - try loading your VG configuration either from the disk itself (using dd if=/dev/sdc3 bs=512 count=255 skip=1) or from the /etc/lvm/backup folder of your former root filesystem (which I understand is on /dev/sdc1) into /etc/lvm/backup/ and re-issuing the vgscan command.

I tried that on both sda3 and sdc3 (as you can see, I have 3 lvm partitions to do this too) and they all result in binary files in the output text file. Ok, correction. There is some lvm meta data in the file, but it's several bytes into the file. I'm looking through the data, but it looks correct. I will keep trying to go through that restore process.

This is expected - both, VG and LV configs is cleartext within binary structures.

I ended up using a slightly modified process than what was outlined here. I ended up making a cfgbackup file from the byte data in the LVM, do a pvcreate, then a vgcfgrestore. After that, it worked. Thanks for the help.

Related Topic