How to Recover RAID – Mount Can’t Read Superblock

mdadm

I had a raid5 array on /dev/md0 created with mdadm and it was working fine for about a year. It consisted of three HDDs of 1TB each. A few days ago there was a power failure and UPS failure. It was not the first time unfortunately.

The OS is on a separate SSD disk (/dev/sda) which is not part of the raid array, so it boots but it cannot mount the array anymore. Sometimes /dev/md0 does not exists at all. Also I did something that might have get the things worse. I ran fsck /dev/sdb -y which wrote many many times on the disk.

I am afraid that I wont recover my files. Can you help me solve this problem?

Thanks.

mount /dev/md0 /mnt/raid5

mount: /dev/md0: can't read superblock

syslog:

Feb 25 15:59:53 pve kernel: [  365.559378] EXT4-fs (md0): unable to read superblock
Feb 25 15:59:53 pve kernel: [  365.560118] EXT4-fs (md0): unable to read superblock
Feb 25 15:59:53 pve kernel: [  365.560216] EXT4-fs (md0): unable to read superblock
Feb 25 15:59:53 pve kernel: [  365.560310] FAT-fs (md0): unable to read boot sector

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] 
unused devices: <none>

fdisk -l

Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x75633c0d

Device     Boot Start        End    Sectors  Size Id Type
/dev/sdb1        2048 1950353407 1950351360  930G fd Linux raid autodetect

Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F397C12B-1549-45EA-97EA-6A41B713B58F

Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 1950353407 1950351360  930G Linux RAID

Disk /dev/sdd: 931.5 GiB, 1000203804160 bytes, 1953523055 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcced27e3

Device     Boot Start        End    Sectors  Size Id Type
/dev/sdd1        2048 1950353407 1950351360  930G fd Linux raid autodetect

sometimes fdisk -l

-bash: /sbin/fdisk: Input/output error

syslog:

Feb 25 16:03:25 pve kernel: [  577.221547] ata1.00: SRST failed (errno=-16)
Feb 25 16:03:25 pve kernel: [  577.232569] ata1.00: reset failed, giving up
Feb 25 16:03:25 pve kernel: [  577.232640] ata1.00: disabled
Feb 25 16:03:25 pve kernel: [  577.232643] ata1.01: disabled
Feb 25 16:03:25 pve kernel: [  577.232658] ata1: EH complete
Feb 25 16:03:25 pve kernel: [  577.232683] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Feb 25 16:03:25 pve kernel: [  577.232697] sd 0:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 05 13 a0 38 00 00 08 00
Feb 25 16:03:25 pve kernel: [  577.232702] blk_update_request: I/O error, dev sda, sector 85172280
Feb 25 16:03:25 pve kernel: [  577.232784] Buffer I/O error on dev dm-6, logical block 9255, lost sync page write
Feb 25 16:03:25 pve kernel: [  577.232928] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Feb 25 16:03:25 pve kernel: [  577.232936] sd 0:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 02 88 6a 10 00 00 68 00
Feb 25 16:03:25 pve kernel: [  577.232941] blk_update_request: I/O error, dev sda, sector 42494480
Feb 25 16:03:25 pve kernel: [  577.232948] EXT4-fs error (device dm-6): kmmpd:176: comm kmmpd-dm-6: Error writing to MMP block

EDIT 1:

sudo mdadm –examine /dev/sdb1

mdadm: No md superblock detected on /dev/sdb1.

sudo mdadm –examine /dev/sdc1

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 34c11bda:11bbb8c9:c4cf5f56:7c38e1c3
           Name : pve:0
  Creation Time : Sun Jun  5 21:06:33 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 1950089216 (929.88 GiB 998.45 GB)
     Array Size : 1950089216 (1859.75 GiB 1996.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : active
    Device UUID : be76ecf7:b0f28a7d:718c3d58:3afae9f7

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Feb 20 14:48:51 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : ffbc1988 - correct
         Events : 2901112

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)

sudo mdadm –examine /dev/sdd1

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x9
     Array UUID : 34c11bda:11bbb8c9:c4cf5f56:7c38e1c3
           Name : pve:0
  Creation Time : Sun Jun  5 21:06:33 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 1950089216 (929.88 GiB 998.45 GB)
     Array Size : 1950089216 (1859.75 GiB 1996.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : active
    Device UUID : 7b9ed6e0:ffad7603:b226e752:355765a8

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Feb 20 14:48:51 2017
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 19b6f3da - correct
         Events : 2901112

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)

Best Answer

Thanks to all I recovered the data.

I ran sudo mdadm --verbose --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 to assemble the array from the two remaining good HDDs and it worked!

Then I formated sdb and re-added it to the array with sudo mdadm --manage /dev/md0 --add /dev/sdb1 and I am going to buy a new one to replace it soon. Also I am looking on backup solutions..

Related Topic