How to recover from – can’t read superblock error – after using lvreduce command (CENTOS7 XFS)

restorexfs

I was attempting to reallocate unused disk space (1.2Tb) from /dev/mapper/centos-home to /dev/centos/root using the steps in How to move disk space from centos-home to centos-root.

After running…

$ umount /dev/mapper/centos-home
$ lvreduce -L 1200G /dev/mapper/centos-home

I'm getting a "can't read superblock" error when i try to remount the drive.

$ mount /dev/mapper/centos-home
mount: /dev/mapper/centos-home: can't read superblock

Before starting, I double-checked to make sure there w enough free (physical space) before running the 'lvreduce' command (there was 2Tb free) – but assume my error was not running a command to first shrink the file system as suggested in LVM logical volume partition corrupted after lvreduce, but also read that this can't be done on XFS systems pr have been unable to locate the specifics.

I tried to recover using,

$ xfs_repair /dev/mapper/centos-home

but the result said

Sorry, could not find valid secondary superblock; Exiting now.

I also tried to restore the size of the LV

$ lvextend -L 1200G /dev/mapper/centos-home

with the result,

New size (307200 extents) matches existing size (307200 extents)

but got the same superblock read error when trying to $ mount the drive.

I'm not sure how deep of a hole I have dug for myself at this point, so here's my question.

What's the best way to recover from this? Or, if I can't recover and mount the damaged drive, do I just delete it and create a new one with the same name? And is this even possible, i.e. do I need to reinstall CentOS? I didn't back up this drive as I haven't been using it. I used the PLESK backup tool so could restore that way, but assume this only works on the /dev/centos/root drive so wouldn't apply here?

I have 4 physical disks set up as RAID5. The other drives and applications are running fine (PLESK lives on /dev/centos/root) and I don't want to damage them.

I'd also like to complete the original task of growing /dev/centos/root.

Here's where things are now….

$ pvscan
  PV /dev/sda3   VG centos          lvm2 [2.04 TiB / 838.63 GiB free]
  Total: 1 [2.04 TiB] / in use: 1 [2.04 TiB] / in no VG: 0 [0   ]

$ vgscan
  Reading volume groups from cache.
  Found volume group "centos" using metadata type lvm2

$ lvscan
  ACTIVE            '/dev/centos/swap' [4.00 GiB] inherit
  ACTIVE            '/dev/centos/home' [1.17 TiB] inherit
  ACTIVE            '/dev/centos/root' [50.00 GiB] inherit


$ pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               centos
  PV Size               2.04 TiB / not usable 0
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              535714
  Free PE               214690
  Allocated PE          321024
  PV UUID               qISA9P-0jFe-gk9I-5uuL-OODW-N3af-OkfJlB

$ vgdisplay
  --- Volume group ---
  VG Name               centos
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               2.04 TiB
  PE Size               4.00 MiB
  Total PE              535714
  Alloc PE / Size       321024 / 1.22 TiB
  Free  PE / Size       214690 / 838.63 GiB
  VG UUID               zzLRUZ-96Vd-5NGr-YsCl-PJ0l-O8Sy-07Dbqu


$ lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                Dxqf6A-Jtyp-Qdhx-1C0u-CS40-L95c-qF4tRU
  LV Write Access        read/write
  LV Creation host, time localhost.localdomain, 2016-12-20 09:00:45 -0500
  LV Status              available
  # open                 2
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/centos/home
  LV Name                home
  VG Name                centos
  LV UUID                9QYG2p-0Srl-AluZ-qsBE-wCQ4-BXyN-0hmTp3
  LV Write Access        read/write
  LV Creation host, time localhost.localdomain, 2016-12-20 09:00:46 -0500
  LV Status              available
  # open                 0
  LV Size                1.17 TiB
  Current LE             307200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                WrqVSs-EkZI-WD7W-AfsL-2PFX-iZLy-8TPjZr
  LV Write Access        read/write
  LV Creation host, time localhost.localdomain, 2016-12-20 09:01:40 -0500
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Best Answer

SmallLoanOF1M, thanks for the suggestion. Because in my case I didn't need to recover the data on the /home logical volume (I appreciate this is an exception for most), I followed the instructions here and simply removed it using the lvremove command.

$ lvremove -v /dev/centos/home

Which returned the remaining free space to the volume group.

I then used the lvextend to extend the /root lv

$ lvextend -L +900G /dev/centos/root

And

$ xfs_growfs /dev/centos/root

To extend the file system in the now larger LV. Next time I install centos7 for a plesk application, I'll be sure to allocate more than 50G to /root, which is the default setting.

1. Create Partitions on sdb

Just as you did create the partitions on sdb. Make sure sdb1 has the same start and end values as sda1. Create sdb2 just as you did earlier (i.e. not the whole rest of the drive)

2. Clone sda1 to sdb1

Remount /dev/sda1 in read-only mode with mount -o remount,ro /boot and clone the partition dd if=/dev/sda1 of=/dev/sdb1.

3. Swap drives and make sure sdb boots

Swap sda and sdb. Install the bootloader on the new sda and make sure the bootloader still loads the kernel when the new sdb (i.e. the original drive) is not connected. Boot will panic because there's no root volume, but you now know that it can boot. Reattach sdb.

3. Extend your Volume Group

Add the new sda2 to your volume group like this: pvcreate /dev/sda2 and vgextend centos /dev/sda2

4. Move over your swap and root volumes

The following commands will move not clone your volumes onto your new drive. The will however stay in the same volume group

pvmove -n swap /dev/sdb2 /dev/sda2
pvmove -n root /dev/sdb2 /dev/sda2

5. Clone your /home

Create a new volume for /home on sda2, mount it somewhere and clone the contents with xfsdump and xfsrestore

lvcreate --size 77,50G --name new_home centos /dev/sda2
mount /dev/centos/new_home /mnt
xfsdump -J /home | xfsrestore -J /mnt

6. Make sure selinux labels got transferred

Check that ls -Z /home and ls -Z /mnt match

7. Rename home volumes

Umount both home volumes and rename them accordingly

umount /home
umount /mnt
lvrename centos/home centos/old_home
lvrename centos/new_home centos/home
mount /home

You have now your cloned smaller /home mounted. The volume centos/old_home is the last one still on sdb2. At this point your system should boot and run without any hassle, but you still need to get rid of sdb.

8. Cleanup

After your have rebooted and made sure everything still works as expected you can remove the old home volume using lvremove centos/old_home. When you look at the output of pvs you should notice that sdb2 should be completely unused.

At this point you can vgreduce centos /dev/sdb2 and pvremove /dev/sdb2 so the "old" drive is no more part of your volume group.

You can now shutdown the box and remove the old drive.