Lvm – centos7 xfs lmv cloning on smaller device

centos7cloninglvmselinuxxfs

I'm going to repropose a question posted on stackoverflow a week ago, I'm sorry if someone found this spam but I still need help to solve this issue. I thought it could be useful also for others.
First of all, I have a server based on Centos7 which run on a non compliant hardware as corporate policy, I just need to support the fault tolerance.

I'm colliding with an apparently simple thing.

I need to put my server behind RAID controller, that's means I need to move my CentOS 7 installation on a device cabled on it (a 3ware controller 9650se-2lp, the computer is an HP business class, DC7600 Convertible Minitower).

My troubles begins with filesystems adopted, is an XFS which can't be reduced by design. I need reduce it because the same disk seen by the controller looks smaller becuase the controller store the firmware in it.

My system is a default installation of CentOS 7 with LVM volumes. After a week of documentation on what and how device mapper works, I ended up I have one volume group called "centos" and the logical volumes called "root", "home" and "swap". Easy.

# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--  148,56g 64,00m
# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  centos   1   3   0 wz--n- 148,56g 64,00m
# lvs
  LV   VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos -wi-ao---- 96,50g
  root centos -wi-ao---- 50,00g
  swap centos -wi-ao----  2,00g

Now I need to reduce my whole layout and I will achieve this reducing the "home" lvm from sda2. I have planned to have a new installation to 130G while now is 148,56G, so I thought to reduce the "home" of 18,56G.

First of all fdisk to the new disk to create the new two volumes, sdb1 and sdb2, After that I have began handling the logical volumes.

# pvcreate /dev/sdb2
  Physical volume "/dev/sdb2" successfully created.
# vgextend centos /dev/sdb2
  Volume group "centos" successfully extended
# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--  148,56g  64,00m
  /dev/sdb2  centos lvm2 a--  130,00g 130,00g

# lvcreate --size 77,50G --name home1 centos
  Logical volume "home1" created.
# mkfs.xfs /dev/centos/home1
meta-data=/dev/centos/home1      isize=512    agcount=4, agsize=5079040 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=20316160, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=9920, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# lvcreate --size 50,00G --name root1 centos
# mkfs.xfs /dev/centos/root1
# lvcreate --size 2G --name swap1 centos
# mkswap /dev/centos/swap1

After that, from the livecd Clonezilla I have cloned the sda1 on sdb1 without rewriting the partition table and from the command line I have mounted the logical volumes to copy the data. I cannot use the simple dd which I know is the best, because dd will copy also the blank space and the new home will be reduced.

# mkdir /mnt/home
# mkdir /mnt/home1
# mount -o ro /dev/centos/home /mnt/home
# mount /dev/centos/home1 /mnt/home1
# cd /mnt/home
# find . -mount -print | cpio -pdm /mnt/home1
5885177 blocks
# cd /
# umount home
# umount home1
# mkdir /mnt/root
# mkdir /mnt/root1
# mount -o ro /dev/centos/root /mnt/root
# mount /dev/centos/root1 /mnt/root1
# cd /mnt/root
# find . -mount -print | cpio -pdm /home/root1
43415372 blocks

Done that, I have powered off the computer, I have uncabled the sda disk and moved the sdb on the first sata port on my motherboard. Now sda is safe from my pudding hands. The best is yet to come.

I have booted again from Clonezilla without copy anything, and I have cleanup and renamed my logical volumes.

# vgreduce --removemissing --force centos
WARNING: Device for PV 0tJwU5-KA41-LEJa-hhFG-SIzx-RGq3-fJNzkm not found or rejected by a filter.
WARNING: Removing partial LV centos/swap.
Logical volume "swap" successfully removed
WARNING: Removing partial LV centos/home.
Logical volume "home" successfully removed
WARNING: Removing partial LV centos/root.
Logical volume "root" successfully removed
Wrote out consistent volume group centos.
# lvrename /dev/centos/root1 /dev/centos/root
Renamed "root1" to "root" in volume group "centos"
# lvrename /dev/centos/home1 /dev/centos/home
Renamed "home1" to "home" in volume group "centos"
# lvrename /dev/centos/swap1 /dev/centos/swap
Renamed "swap1" to "swap" in volume group "centos

Finally I have installed grub2:

# mount /dev/centos/root /mnt
# mount /dev/sda1 /mnt/boot
# mount --bind /dev /mnt/dev &&
mount --bind /dev/pts /mnt/dev/pts &&
mount --bind /proc /mnt/proc &&
mount --bind /sys /mnt/sys
# chroot /mnt
# grub2-install /dev/sda
# grub2-install --recheck /dev/sda
# grub2-mkconfig -o /boot/grub2/grub.cfg

I have fixed a warning on grub2-mkconfig that replace update-grub of grub1 enabling a new service that I never used before lvm2-lvmetad

# grub2-mkconfig -o /boot/grub2/grub.cfg
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
Found linux image: /boot/vmlinuz-3.10.0-514.21.2.el7.centos.plus.x86_64
Found initrd image: /boot/initramfs-3.10.0-514.21.2.el7.centos.plus.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-6f6078afdaf94da59deb321bd428dfdf
Found initrd image: /boot/initramfs-0-rescue-6f6078afdaf94da59deb321bd428dfdf.img
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
done

# systemctl enable lvm2-lvmetad.service
# systemctl enable lvm2-lvmetad.socket
# systemctl start lvm2-lvmetad.service
# systemctl start lvm2-lvmetad.socket

Of course the service start did not work from livecd, I know it.

Once rebooted my system, I got several problem with SELinux. It keep bouncing me back once the login was done, both from remote than local, ssh or not. Of course from livecd is easy set SELinux to permissive, but this is a production server and it hosts services for developing such as mediawiki and git. I can't begin using the cloned drive, put new data on a nearly bugged system, I don't understand what I wrong. I tried reinstalling SELinux related packages, but purely random don't figure out what could be wrong. Am I missing something?

Thanks for the answers,
regards.

Best Answer

All of the following steps (except the bootloader installation) will be done on the running system.

1. Create Partitions on sdb

Just as you did create the partitions on sdb. Make sure sdb1 has the same start and end values as sda1. Create sdb2 just as you did earlier (i.e. not the whole rest of the drive)

2. Clone sda1 to sdb1

Remount /dev/sda1 in read-only mode with mount -o remount,ro /boot and clone the partition dd if=/dev/sda1 of=/dev/sdb1.

3. Swap drives and make sure sdb boots

Swap sda and sdb. Install the bootloader on the new sda and make sure the bootloader still loads the kernel when the new sdb (i.e. the original drive) is not connected. Boot will panic because there's no root volume, but you now know that it can boot. Reattach sdb.

3. Extend your Volume Group

Add the new sda2 to your volume group like this: pvcreate /dev/sda2 and vgextend centos /dev/sda2

4. Move over your swap and root volumes

The following commands will move not clone your volumes onto your new drive. The will however stay in the same volume group

pvmove -n swap /dev/sdb2 /dev/sda2
pvmove -n root /dev/sdb2 /dev/sda2

5. Clone your /home

Create a new volume for /home on sda2, mount it somewhere and clone the contents with xfsdump and xfsrestore

lvcreate --size 77,50G --name new_home centos /dev/sda2
mount /dev/centos/new_home /mnt
xfsdump -J /home | xfsrestore -J /mnt

6. Make sure selinux labels got transferred

Check that ls -Z /home and ls -Z /mnt match

7. Rename home volumes

Umount both home volumes and rename them accordingly

umount /home
umount /mnt
lvrename centos/home centos/old_home
lvrename centos/new_home centos/home
mount /home

You have now your cloned smaller /home mounted. The volume centos/old_home is the last one still on sdb2. At this point your system should boot and run without any hassle, but you still need to get rid of sdb.

8. Cleanup

After your have rebooted and made sure everything still works as expected you can remove the old home volume using lvremove centos/old_home. When you look at the output of pvs you should notice that sdb2 should be completely unused.

At this point you can vgreduce centos /dev/sdb2 and pvremove /dev/sdb2 so the "old" drive is no more part of your volume group.

You can now shutdown the box and remove the old drive.

Related Solutions

Linux – Logical Volumes Inactive at Boot Time

So I managed to solve this eventually. There is a problem (bug) with detecting logical volumes, which is some sort of race condition (maybe in my case regarding the fact that this happens inside KVM). This is covered in the following discussion. In my particular case (Debian Squeeze ) the solution is as follows:

backup the script /usr/share/initramfs-tools/scripts/local-top/lvm2
apply the patch from mentioned bug report
run update-initramfs -u

This helped me, hope it'll help others (strangely, this is not part of mainstream yet).

Link to patch: _http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=lvm2_wait-lvm.patch;att=1;bug=568838

Below is a copy for posterity.

--- /usr/share/initramfs-tools/scripts/local-top/lvm2 2009-08-17 19:28:09.000000000 +0200
+++ /usr/share/initramfs-tools/scripts/local-top/lvm2 2010-02-19 23:22:14.000000000 +0100
@@ -45,12 +45,30 @@

  eval $(dmsetup splitname --nameprefixes --noheadings --rows "$dev")

- if [ "$DM_VG_NAME" ] && [ "$DM_LV_NAME" ]; then
-   lvm lvchange -aly --ignorelockingfailure "$DM_VG_NAME/$DM_LV_NAME"
-   rc=$?
-   if [ $rc = 5 ]; then
-     echo "Unable to find LVM volume $DM_VG_NAME/$DM_LV_NAME"
-   fi
+ # Make sure that we have non-empty volume group and logical volume
+ if [ -z "$DM_VG_NAME" ] || [ -z "$DM_LV_NAME" ]; then
+   return 1
+ fi
+
+ # If the logical volume hasn't shown up yet, give it a little while
+ # to deal with LVM on removable devices (inspired from scripts/local)
+ fulldev="/dev/$DM_VG_NAME/$DM_LV_NAME"
+ if [ -z "`lvm lvscan -a --ignorelockingfailure |grep $fulldev`" ]; then
+   # Use default root delay
+   slumber=$(( ${ROOTDELAY:-180} * 10 ))
+
+   while [ -z "`lvm lvscan -a --ignorelockingfailure |grep $fulldev`" ]; do
+     /bin/sleep 0.1
+     slumber=$(( ${slumber} - 1 ))
+     [ ${slumber} -gt 0 ] || break
+   done
+ fi
+
+ # Activate logical volume
+ lvm lvchange -aly --ignorelockingfailure "$DM_VG_NAME/$DM_LV_NAME"
+ rc=$?
+ if [ $rc = 5 ]; then
+   echo "Unable to find LVM volume $DM_VG_NAME/$DM_LV_NAME"
  fi
 }

Linux – XFS: no space left on device, (but I have 850GB available!)

I was able to resolve the issue in the following way:

umount [the mountpoint]
mount /dev/data_vg/data_lv -o inode64 [the mountpoint]

Apparently, the default (32-bit inodes?) xfs will store all inodes in the first 1TB portion of the disk. This means that if the first 1TB is full, then you'll run into no space on disk errors even if it appears you have plenty of space/inodes available. By adding the inode64 option, the nodes can be stored anywhere on disk, if I understand correctly.

Source: the XFS FAQ.