This problem is on a kvm/qemu hypervisor with CentOS 7, where we host multiple virtual machines.
The Disk /dev/sda( 1918.2 GB) is a raid6 LSI MEGARAID.
[root@mia3 ~]# fdisk -l
Disk /dev/sda: 1918.2 GB, 1918239768576 bytes, 3746562048 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x51549cd4
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 135243775 67108864 82 Linux swap / Solaris
/dev/sda3 135243776 449816575 157286400 83 Linux
/dev/sda4 449816576 3746562047 1648372736 8e Linux LVM
This is our disk configuration:
sda1 – /boot,
sda2 – swap,
sda3 – / (root),
sda4 – PV for our LVM with thin provisioning for our VMs.
After a power failure, our data on the root partition got corrupted. We had to reinstall the OS in order to bring up the server. We didn't touch the /dev/sda4 partition.
Before that, we mounted the root partition in rescue mode and made a backup for the /etc/lvm/ folder.
After reinstalling Centos we tried to recover LVM using vgcfgrestore, but we got the error:
[root@mia3 ~]# vgcfgrestore -f /etc/lvm/backup/vg_kvm vg_kvm -v
No devices found.
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
Consider using option --force to restore Volume Group vg_kvm with thin volumes.
Restore failed.
Scanning all devices to initialize lvmetad.
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
[root@mia3 ~]#
So we tried to readd the PV and ovewrite the metadata, but without success:
[root@mia3 ~]# pvcreate -f /dev/sda4 --restorefile /root/lvm/backup/vg_kvm --uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT -vv
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]
global/lvmetad_update_wait_time not found in config: defaulting to 10
devices/filter not found in config: defaulting to filter = [ "a|.*/|" ]
devices/cache not found in config: defaulting to /etc/lvm/cache/.cache
metadata/record_lvs_history not found in config: defaulting to 0
File-based locking selected.
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
Stack vg_kvm/lvpool:0[0] on LV vg_kvm/lvpool_tdata:0.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tdata.
Adding vg_kvm/lvpool:0 as an user of vg_kvm/lvpool_tmeta.
Adding vg_kvm/vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12038-dm5mulwohkvz5ara-kccgnknpjdi11tdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12039-d1lnb9qcyjaqmodk-mnl22loaflo24kpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12040-daejvderic4uvccj-wjnydccgzohyaekq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12046-dswief4uk4gthxa1-ie5s7up6vbf37p68:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12056-du7hvvsp3lw7jgja-uap8l1jpe29llrpu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12080-dwydbngcjptnnryn-hzz0jimwfl6yau2x:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12094-d8cmko7aio2f1fyo-warkvewdzfeedme5:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12097-dllp7oepp4homdj9-jm6n2itmvy29k9xf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12098-dcj6mnnidej8c80u-r3bkk3pfu8g9sf3o:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12103-dwtyfq1quemcij56-4rwivmmtzxwk1ydb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12106-dngrxbyoapwac5sm-bstrvsbxzrztefob:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12107-dq3h0wgnzkasoles-hv12r8qmyobojubp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12109-ddq6cmtow9qwuaka-v9a1omyljy0xujk4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12110-dvaaktx2bng99yl9-f5xh28mlg7u3mo5r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12111-db0wlufykbcbb2mp-dxhgxxmivthvqwwp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12112-dyfnblgyajfpidqs-ny0xpnxhzfzwc1db:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12116-dowxexkxa0ygsgki-e4dih6xcnybtdgx2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12117-dwd0weu8ig1dvukx-fly3n8dyseklhntv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12118-dz9hbvnbtsihufm5-paqhexr1tuld8bnd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12119-dof5n90vez6smo1j-h1xmccyzl7qotery:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12120-d9q0y85z4shpw64o-6zx5tmxhajpcxtkv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12121-dac4vuig8akyryd0-0e4rf2snm04r5mdy:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12122-d99yza6ssbhxxkru-16ptyp8btexq6wkb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12124-d4ud5tu2ueajr0vc-r9pv77cjusnjue21:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12126-dqjmtfzubhqvheij-blecac71kpwsmy62:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12127-d5sgcmdodjaqtvp9-oqbroobow5tzn0va:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12130-dtf7rf4suf8bov9t-8lb0agstvcpann2e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12131-dwkveordvertdam3-3g4qwtf5ydhy0hug:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12134-dgpxbewt3xlkxlqb-j5ad04pt2fndqqmw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12135-d55krjg3dotss3e4-dwbbjmo0shwwwzv2:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12138-dyz9yl6swxbaxfrj-1pgri05suzqyagqq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12140-dykgcwqaauznoxrm-4ibft4womwuvpwkx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12142-dwtlkzoke0a05u1o-rqxjogzuvjvfttxs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12144-dn3gyu3uroppwgzg-wraejm2ynwnmtmd4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12162-dalw90et3l9vbath-eo5womktdromb5sn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12211-dyyizuuag0mz4eok-je7du6slil1jlfzp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12329-d5xmlj2iqsuekgkm-cg6qplkwk48yiyeo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12333-drqvvb7fktr4n52b-estbavvooqdljpvo:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12334-dbdf0lvefbgpqbxa-bmpnzainjdfvuavv:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12339-d3yfeg1upsfoxbd3-nbxbccudjdtcgztt:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12354-d1ctzl88dwv9znri-pzfp6weunhpdhvgn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12361-dbug4gr7r8mbsnva-yamgras92xuqin66:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12088-dqphwqqeuwxtmm7y-4rurc6jm0lvcj57q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12089-dbhimrcqvjkaxvxc-3vpljlh2xkojhb9e:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12090-dndlzsmpj8qgm2gb-wm4sg3d6n2cq3tpm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12091-dyyxcz5cf4w4cu6z-gja2kph6egpjvyzh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12092-dnllln6x7uud75ss-um9zelymhc04ohd6:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12442-dyf2cgwdgaluli0s-cahwc74lrstt61gl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12456-dmiuo7guhru4snsf-yt55qun0wx5mtfkg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12087-dxbtb8w1g3tx5akk-bdqcsnlathemkrig:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12234-dq82wyoapvxvh7zz-maaekbfd7h9ixks8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12588-das5kos8qt1ziktl-aky0pxwg8ecv9b48:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12771-djaikmniei1vxia5-jhqasdwc763s3pmb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12784-d3k6aplejbvjrlcw-uwo81r0wkmeiptso:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12139-dpnhdmg2odfabvfn-zqgiw42wfsgbbkrc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12137-dxtiqptvdwcrrcwa-22fkxorf6hl44evf:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12866-d37htmu48aopiafv-sli9m0lt3dz0yeyl:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12096-dz9pfno05wqq8vbg-sq8i5na5z8lrnm6n:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12363-dtlh9w3tza99quoj-dlicfndz6e3pbfu0:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12890-dmaaydjldw3c13jk-iskdygh5pczveo3r:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12320-dqmecms9nep6muns-xmmkziafa8mz1hta:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12936-dhdfa36ovcdzfafp-ultnlmom4scugsw1:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12964-dctbp9xgo2gl8w6i-kbw9cinhoieea8av:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12984-dwqjvkqc3aznhb4o-criyexpxvdndoscq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12143-d3s1toagczaiq4ma-oupwnk3eg27r41ze:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13019-d5yrsi5a58bdzalc-i6mbum8zthk2arhs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13022-djszrasae5w7xegh-ehqsutbhrwd7qgsi:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13027-dq6pg99jd2fphljg-u15x8sonbwukaree:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12041-dlsmtmfa6fp4wcuc-bfhftkdhwfqa7sqb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13039-djtjimojqi3iaufp-csbinqtsvebzzmqx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13061-duocptl9qdtvwubp-feryqeejht4j24ql:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13064-dexwkomdh62by80s-aymbuetu6p0btn4d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13076-drd4nrhy4u7fq1tn-f1kt8ubk2jch4ol3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13082-ds9gjikqyg9nftua-rrhk4dlrr8cp3la7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12759-du4yducr4ghi8eun-bzivqxabwaltaddm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12123-dzb9rktypp0jqinx-adtlzed02wpccxoa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13118-d0lj9gpvqfmox63i-9jstjcjkmemszrla:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13121-du0r39n6wvnrclox-bviz76ig9vvexota:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13133-d2fk6lg6bsppbkcs-hwk5bmsvqxa0y0bs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13204-dl6hzsmswetitsb4-bais6lpjbbhfkexn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13230-dtiyrmijg8mde5xy-vobukov1qxg8bx5j:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13254-d3sdhrrgcvbwrsyp-yxaozxmt3clqzxxc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13271-drutw4ihq774xhs6-ecpwfmzcrqy0vvsm:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13266-dmuj5cfeu2hrgfxw-2tpflycnwoedyidb:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13380-d99ciziibe7rt3es-7fkh56myy79oushw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13487-d30yxx4wgfxcb9te-chc2bxojs7tvcwz3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13502-dc2oivdodzo94jj3-1qmpba9ti4zzbfeg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13277-deudjco25w9cd3rv-re1ha5qkozses9i3:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12208-dfqhoqmtuwzktwqm-n9s5daovhtnyvyeh:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13481-dthcwgpolombhfed-dkmesvsvmt1vem0d:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13585-ddvwhm2vcqpq66uy-1brs98bctgr6bcq9:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13586-dadacaexxn8nkuy3-fyx8zp5mursbqhns:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13587-dc0fmskwdbkmz7mg-ra53wgfksiv7tsqk:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13569-dk6v0pyvz8eoohl5-qyndiwn8zriir75s:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13603-d66mz2amabmd63fj-rkcqmtvfnlfmywou:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13613-dykuo32mnkgf9nce-hhkhn118dhsedoc8:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13623-dksoizdtoglihaax-23o0zltutliyoo2q:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13629-dxbpqqernfvufxrz-iv6jujdxm78hxjbn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13610-drrehnh36oipacdf-frnumdhlt823mntg:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13654-dmlnhhqilztxff5a-4gawkysgtkze4st7:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13669-dk3ukgidvj0l4x4x-9d3vfao5qejzecdn:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13672-d1r1608gu96b0ujo-nn7ejmxkbfce40m4:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13689-dhpzh8w7jlwi0zza-aqizb6tbeuk9k1ca:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12477-dejmu0gihtllnj97-gaxwgicndwp25unx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13710-dv2bktawped5fz44-0bkdtrkp8tk2vynu:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12125-d5d9pebts1rue7jt-vwnkapewjkfszked:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13718-dmimkn9h1ax3t4am-8tdnpznya2reek8g:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13545-dzaq2rjqfiyf5azl-kw35xomndhltjnrs:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13723-daleipfjwu7zm7aj-s8jo6yhxwz4efl2y:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13726-dpdqvmnzkvxkiz4y-cfudhed6m12hnlmj:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12136-d0qvvamyx5qq1g6o-awtlh6pbotvlacyc:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13258-du2kuwgkwhypcfzz-i6h00rd0exzw4b69:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13755-dqglblzy3akyohph-wtev5tlrs5lkkejd:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13758-dhfrybljxildcky1-ktotkou3mpvxfjnq:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13759-dxqbq7phpjgoa1ed-dwtly1dkuorvlyli:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13762-di54bo3us4p8gefa-34s2uhcye1rbdbvw:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13767-dwtsyqxhqbp8fa9w-us8xvwm9q6nqkmzx:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13768-dadnlkrau2cwhlbd-cj8liz4hfolpmjfa:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13769-dt4nwlcmpfqyenob-jvpfd5jut8kvcm8u:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv13781-dvckr0uqfwrsvmc6-zklf1olhlhzzfppp:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12133-dp55dttaolpbd6n8-rlhoezgituqkyrov:0 as an user of vg_kvm/lvpool.
Adding vg_kvm/vsv12042-dnncyzlyhfxyaxkt-metx0ei0zlz3d2sn:0 as an user of vg_kvm/lvpool.
metadata/pvmetadataignore not found in config: defaulting to 0
metadata/pvmetadatasize not found in config: defaulting to 255
metadata/pvmetadatacopies not found in config: defaulting to 1
report/output_format not found in config: defaulting to basic
log/report_command_log not found in config: defaulting to 0
Locking /run/lock/lvm/P_orphans WB
/dev/sda4: size is 3296745472 sectors
/dev/sda4: using cached size 3296745472 sectors
/dev/sda: size is 3746562048 sectors
/dev/sda1: size is 1024000 sectors
/dev/sda1: using cached size 1024000 sectors
/dev/sda2: size is 134217728 sectors
/dev/sda2: using cached size 134217728 sectors
/dev/sda3: size is 314572800 sectors
/dev/sda3: using cached size 314572800 sectors
/dev/sda4: using cached size 3296745472 sectors
/dev/sda4: using cached size 3296745472 sectors
Locking #orphans_lvm1 already done
Reading VG #orphans_lvm1
Locking #orphans_pool already done
Reading VG #orphans_pool
Locking #orphans_lvm2 already done
Reading VG #orphans_lvm2
Processing device /dev/sda1.
Processing device /dev/sda2.
Processing device /dev/sda3.
Processing device /dev/sda4.
/dev/sda4: lvm2 label detected at sector 1
/dev/sda4: Checksum error at offset 808960
Couldn't read volume group metadata from /dev/sda4.
Metadata location on /dev/sda4 at 808960 has invalid summary for VG.
Failed to read metadata summary from /dev/sda4
Failed to scan VG from /dev/sda4
Unlocking /run/lock/lvm/P_orphans
[root@mia3 ~]#
Note the error:
Couldn't find device with uuid HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT.
The PV (/dev/sda4) has the same UUID:
[root@mia3 ~]# blkid
/dev/sda1: LABEL="boot" UUID="087755bd-9434-4597-953b-c056fd9cb285" TYPE="xfs"
/dev/sda2: LABEL="swap" UUID="c7fd2030-0441-4a4e-8462-0aeffc272bfa" TYPE="swap"
/dev/sda3: LABEL="root" UUID="3ee19cf6-87b5-40b5-a7dd-13d507efa734" TYPE="xfs"
/dev/sda4: UUID="HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT" TYPE="LVM2_member"
[root@mia3 ~]#
Here is a small part of the content for /etc/lvm/backup/vg_kvm:
vg_kvm {
id = "i3usZz-vdt5-9yaU-Kbcl-XDWJ-OFr1-UhZRBa"
seqno = 1349
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0
physical_volumes {
pv0 {
id = "HNB35U-ilI4-WWAK-1Q4D-Netj-t68p-Po4vCT"
device = "/dev/sda4" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 3296745472 # 1.53517 Terabytes
pe_start = 2048
pe_count = 402434 # 1.53516 Terabytes
}
}
logical_volumes {
lvpool {
id = "DHqs7X-95DA-ADNm-WTd9-cXDf-HR5a-MoHLRx"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575901618 # 2019-12-09 16:26:58 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 401081 # 1.53 Terabytes
type = "thin-pool"
metadata = "lvpool_tmeta"
pool = "lvpool_tdata"
transaction_id = 667
chunk_size = 2048 # 1024 Kilobytes
discards = "passdown"
zero_new_blocks = 1
}
}
vsv11999-dm9oryxunpw8po9p-1xwrkrotul6i5k3g {
id = "vBco77-gPno-S0cu-QZdx-9Nnk-twXR-cBBAzW"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575902920 # 2019-12-09 16:48:40 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 6400 # 25 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 0
device_id = 1
}
}
vsv12001-dmzdbquhkhrydlnf-kqjcbeyvz1c5xwun {
id = "xqDzS2-iHXO-2zEi-LRHO-0FDR-hvcM-oRdFeD"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575904408 # 2019-12-09 17:13:28 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 1
device_id = 2
}
}
vsv12034-dnadddzaxsxoq2rx-iwxjz6wr2vmqhsj3 {
id = "VIDuPN-SN4Z-V1qm-gzI9-sSnV-WGcp-NBvgUa"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575986509 # 2019-12-10 16:01:49 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 6
device_id = 5
}
}
vsv12036-dbgt9zilw2tg9sqd-c0xa9h1uxigjuqg9 {
id = "XnnwPm-e3jE-VU9W-7QIz-3Sz5-HDSM-sEe7NY"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575988785 # 2019-12-10 16:39:45 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 8
device_id = 7
}
}
vsv12037-dctrjirji3agbl9l-dtzyf8jrb8lip0m7 {
id = "ysHM90-Vmof-Mrwj-Nb1s-HPER-g3v5-hdutBz"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 1575989058 # 2019-12-10 16:44:18 +0200
creation_host = "mia3"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 3840 # 15 Gigabytes
type = "thin"
thin_pool = "lvpool"
transaction_id = 9
device_id = 8
}
}
All the data (root partitions for our VMs) are still there, we have checked with disk recovery tools like testdisk, but we can't recover the lvm.
What are we doing wrong? Is there a possibility of recovering?
#
UPDATE
Using Centos6 CD and rescue mode I have managed to restore the PV and VG. However, I can't reactivate the VG and LV pool, we're getting the error:
Executing: /usr/sbin/thin_check -q --clear-needs-check-flag /dev/mapper/vg_kvm-lvpool_tmeta
/usr/sbin/thin_check failed: 1
Check of pool vg_kvm/lvpool failed (status:1). Manual repair required!
Best Answer
It appears Red Hat 7 and 8 have similar issues when trying to use
vgcfgrestore: The pvcreate command fails with "Metadata location has invalid summary for VG" error . CentOS shares a base with Red Hat, so...One answer there was "We have a solution for this problem upstream and in RHEL8, Unfortunately, it is too invasive to consider back porting to RHEL7 at this stage." Another person in that bug said they fixed it by booting Red Hat 6, and then
vgcfgrestoreworked.Perhaps you can boot a new sysrescue CD from a USB stick and do it?
And as I said in a comment, either backup the whole volume with
dd, or at least do something likedd if=/dev/sda4 of=/somewere/backup-first100MB.img bs=1M count=100so that you have the header info.