Linux – How to re-assemble (or recover data from) a mdadm RAID5 array

data-recoverylinuxmdadmsoftware-raid

I'm trying to recover whatever data I can from a RAID5 array of a failed NAS.

The disks were physically moved to another server that so they could be imaged,
but Knoppix "helpfully" detected and recovered the array on boot:-

[  677.791667] md: md0 stopped.
[  677.793443] md: bind<sdb>
[  677.793580] md: bind<sdc>
[  677.793866] md: bind<sdd>
[  677.794004] md: bind<sda>
[  678.671014] md: raid6 personality registered for level 6
[  678.671017] md: raid5 personality registered for level 5
[  678.671019] md: raid4 personality registered for level 4
[  678.671446] md/raid:md0: device sda operational as raid disk 0
[  678.671449] md/raid:md0: device sdc operational as raid disk 2
[  678.671452] md/raid:md0: device sdb operational as raid disk 1
[  678.671840] md/raid:md0: allocated 4222kB
[  678.671874] md/raid:md0: raid level 5 active with 3 out of 4 devices, algorithm 2
[  678.671877] RAID conf printout:
[  678.671879]  --- level:5 rd:4 wd:3
[  678.671881]  disk 0, o:1, dev:sda
[  678.671883]  disk 1, o:1, dev:sdb
[  678.671886]  disk 2, o:1, dev:sdc
[  678.671888]  disk 3, o:1, dev:sdd
[  678.671923] md0: detected capacity change from 0 to 4500901330944
  ^ RAID conf printout repeated
  ^ RAID conf printout repeated
[  678.679071] md: recovery of RAID array md0
[  678.679074] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  678.679077] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  678.679083] md: using 128k window, over a total of 1465137152 blocks.
[  678.679086] md: resuming recovery of md0 from checkpoint.
[  678.694387]  md0: unknown partition table
[18209.608703] md: md0: recovery done
[18209.690600] RAID conf printout:
[18209.690605]  --- level:5 rd:4 wd:4
[18209.690607]  disk 0, o:1, dev:sda
[18209.690610]  disk 1, o:1, dev:sdb
[18209.690612]  disk 2, o:1, dev:sdc
[18209.690614]  disk 3, o:1, dev:sdd

Afterwards, the array looks healthy, with /proc/mdstat showing:-

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdd[4] sdc[2] sdb[1]
      4395411456 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

However, I'm unable to mount /dev/md0 directly or find any logical volumes.

Using dd to extract the first 512 bytes of /dev/md0 gets me the partial
contents of a file that was on the array instead of the MBR, so I suspect
this is because the array was assembled in the wrong order.

If correct, how can I re-assemble the array?

If not, what is the best way to try and recover data from the array?

mdadm -D /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Sun Aug  7 17:25:36 2011
     Raid Level : raid5
     Array Size : 4395411456 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 1465137152 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Fri Aug 12 21:54:53 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : Microknoppix:1  (local to host Microknoppix)
           UUID : 37b79cab:d800da6a:eec72e0d:6547b6ef
         Events : 19

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       4       8       48        3      active sync   /dev/sdd

mdadm -E /dev/sd?

/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 37b79cab:d800da6a:eec72e0d:6547b6ef
           Name : Microknoppix:1  (local to host Microknoppix)
  Creation Time : Sun Aug  7 17:25:36 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 8790822912 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 42c9006b:1405120d:edf4d6da:e707f2dd

    Update Time : Mon Aug 15 02:19:35 2011
       Checksum : 40b694de - correct
         Events : 19

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 37b79cab:d800da6a:eec72e0d:6547b6ef
           Name : Microknoppix:1  (local to host Microknoppix)
  Creation Time : Sun Aug  7 17:25:36 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 8790822912 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 39fc5928:11ecb5b6:c3c088e6:2833b7ea

    Update Time : Mon Aug 15 02:19:35 2011
       Checksum : c02aa5ea - correct
         Events : 19

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 37b79cab:d800da6a:eec72e0d:6547b6ef
           Name : Microknoppix:1  (local to host Microknoppix)
  Creation Time : Sun Aug  7 17:25:36 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 8790822912 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 90c64aa7:82b06090:fe9363e1:380dc0da

    Update Time : Mon Aug 15 02:19:35 2011
       Checksum : 3a9e1ff - correct
         Events : 19

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 37b79cab:d800da6a:eec72e0d:6547b6ef
           Name : Microknoppix:1  (local to host Microknoppix)
  Creation Time : Sun Aug  7 17:25:36 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 8790822912 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274304 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : fab3ec51:50f9bbc6:7ac51700:54195903

    Update Time : Mon Aug 15 02:19:35 2011
       Checksum : 2bf455cf - correct
         Events : 19

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)

fdisk -l

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x894e894e

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      267350  2147483647+  ee  GPT

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/md0: 4500.9 GB, 4500901330944 bytes
2 heads, 4 sectors/track, 1098852864 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1572864 bytes
Disk identifier: 0x312e312f

Disk /dev/md0 doesn't contain a valid partition table

Best Answer

I find it quite hard to believe that mdadm would assemble drives in the wrong order; it's metadata isn't that bad. I'm wondering what should be on the RAID array, and what dmesg had to say about matters when you tried (and failed) to mount the array.

If your surmise is correct about the array actually being out-of-order is correct, I'd unplug the disks and either give up on the data, or fork out money for a professional data-recovery place to try and put it back together. There's a chance you'll be able to do it, but there's also a chance that you'll make a mess of it and make the data harder to recover for the pros, and what is your time worth to basically learn data recovery on the fly?

For your pennance, say 10 "RAID is not a backup"s and 5 "Our Linus"es...

Related Topic