Ubuntu – sed fix for BusyBox – boot fail unable to mount LVM

bootfilterlvmsedUbuntu

BusyBox why u no vi?!

Summary:

Ubuntu 14.04 Server after upgrade first boot, only boots to BusyBox v1.21.1 initramfs with "gave up waiting for root device" due to missing /dev/mapper/xxx-root as unable to find/mount LVM VG due to incorrect exclusion filter in lvm.conf

Scenario:

only have the single kernel 3.11.0-17 (yes, no fallback!), both normal and recovery go to BusyBox
only ssh access, or IP KVM (system in remote CoLo in DC over 10,000 miles away)
no access to bootable media as IP KVM does not load virtual media
default initramfs configuration (attempt to use vi shows vi: not found)
LVM PV should be on something like /dev/sda5 (assume this as have many systems with same
config and that is what the others are set as)

Detail:

100% sure issue is incorrect LVM filter (configuring OpenStack Cinder loopback device for LVM poorly), consequently is not scanning for LVM VG. Known working should be default in lvm.conf:

filter = [ "a/.*/" ]

However this is commented out and instead have in lvm.conf:

filter = [ "a/sda1/", "a/sdb/", "r/.*/"]

If I can make the change in lvm.conf by searching for incorrect string and replacing line with correct string in lvm.conf then I believe LVM VG will boot correctly.

Attempted Solution:

initramfs has no editing (vi, vim, nano, etc), only sed
have tried testing sed regex, sadly this is incredibly difficult to crack for this regex noob. I have so far:

sed -i '/filter = [ "a/sda1/", "a/sdb/", "r/.*/"]/c\filter = [ "a/.*/" ]' /etc/lvm/lvm.conf

which makes no change to a test file. I have trouble comprehending the complexities of slashes contained within the sed search/replace string in conjunction with the sed command slash syntax, maybe someone can assist here? If I can crack the sed command regex I'm fairly sure the kernel should be able to boot into LVM.

Alternatively is there a force/manual mount I could use (from grub edit or initramfs), despite the incorrect lvm.conf filter, what is this?

As a fallback, I could go down the path of MaaS (Ubuntu's web based PXE/tftp image booting), which I have configured on another server in the DC, but have not yet tested, or used. I am happy to try to blast a new Ubuntu 14.04 Server onto the non-booting system using MaaS.

Correctly editing lvm.conf by cracking sed first in BusyBox is more desired. I'd like to understand this for future, or any other suggestions? Edit at grub, manual LVM VG boot, or any other suggestions that help are appreciated. Thank you.

Follow up edit:

(not sure on serverfault best practice here, whether I should answer my own question or make this edit, anyway..)

A big thank you to mvillar for the sed guidance. OK, here is the final synatax and steps that saved my bacon from an unbootable system (only to BusyBox) due to an error in lvm.conf filter configuration. I first used the following sed command to double check for correct syntax before and after change to '# print only lines which match regular expression (emulates "grep")':

sed -n '/loop/p' /etc/lvm/lvm.conf

Source: http://www.catonmat.net/blog/wp-content/uploads/2008/09/sed1line.txt

So I discovered the actual incorrect line I had in lvm.conf (differing from what I originally stated):

filter = [ "a/sdc5*/","a/loop2*/", "r/.*/" ]

Note: I believe the cause of the booting error was incorrect syntax in the original lvm.conf. I wanted: filter to accept sda5 and loop2 and reject all others. I was incorrectly stating: sdc5 and possibly missing space after comma

sed regex that worked to replace incorrect syntax with 'accept all' filter in lvm.conf:

sed -i "s/filter \= \[ \"a\/sdc5\*\/\",\"a\/loop2\*\/\", \"r\/\.\*\/\" \]/filter = [ \"a\/.\*\/\" ]/" /etc/lvm/lvm.conf

Lessons learned:
+ escape not needed for spaces, needed for literal string forward slash, asterix and quotes
+ default Ubuntu 14.04 does NOT include vi or tail, despite them both listed as BusyBox commands in upstream, something awry here with Ubuntu BusyBox default build. I will in future check that all my systems include AT LEAST vi and tail in BusyBox builds!
+ sed is at first complicated to understand, but a really useful and powerful tool

Steps to view LVM status, recreate LVM vg and lv and boot from BusyBox after sed and checking:

(initramfs) lvm
lvm> pvdisplay
lvm> vgdisplay
lvm> lvdisplay
lvm> vgchange -a y
lvm> exit
(initramfs) exit

Source: http://ubuntuforums.org/showthread.php?t=1898901

Ubuntu then continued normal boot with active root lv and went straight to login. Awesome!

Best Answer

According to the docs, busybox 1.12.1 has vi.

Anyway, here's your sed regex:

sed -i "s/filter\ \=\ \[\ \"a\/loop2\/\",\ \"r\/\.\*\/\"\ \]/filter = [ \"a\/.*\/\" ]/" /etc/lvm/lvm.conf

Related Solutions

Linux – Rescue disk is unable to see the lvm physical volumes

vgscan -vvvv

will give you very extensive output about why vgscan considers any specific volume being part of a volume group. You also could run pvs -a to see a summary of your physical volumes alongside with volume group assignments.

vgscan -vvvv output for one of the partitions:

Opened /dev/sdc3 RO /dev/sdc3: size is 3772817055 sectors 
Closed /dev/sdc3 /dev/sdc3: size is 3772817055 sectors 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
Closed /dev/sdc3 Using /dev/sdc3 
Opened /dev/sdc3 RO O_DIRECT /dev/sdc3: block size is 512 bytes 
/dev/sdc3: No label detected
Closed /dev/sdc3

pvs -a didn't reveal anything. All physical volumes listed without a volume group assignment

"no label detected" sounds pretty sad. You are sure that it is a LVM2 partition and not, say, a partition used by md-raid? You could check using mdadm --examine /dev/sdc3. And please post fdisk -l /dev/sdc

Yes I'm sure it is an LVM2 partition. The mdadm command gives "No md superblock detectd on /dev/sdc3" The fdisk says /dev/sdc3 is a Linux LVM partition.

Ah, then you will be in the lucky position (irony here, sorry) to try LVM recovery due to presumably damaged data structures. There is a howto about LVM recovery which might give you a starting point - try loading your VG configuration either from the disk itself (using dd if=/dev/sdc3 bs=512 count=255 skip=1) or from the /etc/lvm/backup folder of your former root filesystem (which I understand is on /dev/sdc1) into /etc/lvm/backup/ and re-issuing the vgscan command.

I tried that on both sda3 and sdc3 (as you can see, I have 3 lvm partitions to do this too) and they all result in binary files in the output text file. Ok, correction. There is some lvm meta data in the file, but it's several bytes into the file. I'm looking through the data, but it looks correct. I will keep trying to go through that restore process.

This is expected - both, VG and LV configs is cleartext within binary structures.

I ended up using a slightly modified process than what was outlined here. I ended up making a cfgbackup file from the byte data in the LVM, do a pvcreate, then a vgcfgrestore. After that, it worked. Thanks for the help.

Ubuntu – Grub2 reports physical volume pv0 not found when probing/installing

It sounds like you didn't reboot between upgrading LVM2/GRUB and the kernel.

You will want to "pin" an old grub version long enough to reboot your system into a current kernel version. At that point you should be good to unpin, upgrade grub and everything should be kosher.

Add this to /etc/apt/preferences:

# Force grub to 1.98+20100804-5ubuntu3
Package: grub-pc
Pin: release n=maverick
Pin-Priority: 1000

Now

 apt-get update && apt-get -s dist-upgrade

Assuming you get a line like Inst grub-pc [ 1.98+20100804-5ubuntu3] .... (and the resulting dependency shuffling doesn't break your system... Watch for things like libc downgrades) you can apt-get dist-upgrade to force grub back to sanity. I would suggest running update-grub one more time to make sure you're not walking into an un-bootable situation.

Let me know how it goes :)

Best Answer

Related Solutions

Linux – Rescue disk is unable to see the lvm physical volumes

Ubuntu – Grub2 reports physical volume pv0 not found when probing/installing

Related Topic