I'm a bad person and haven't backed up my RAID elsewhere. I now have a RAID10 array that won't assemble, and I'm hopeful I can save it. Here's the details:
I have five hard drives set up as RAID10 (4+1 Spare). For unknown reasons, two failed and my spare was broken, and now I can't seem to reassemble them.
I have some information below, notably the output of mdstat after a reboot [1], the output of stopping the array and attempting to reassemble it [2] and detailed information from mdadm -E [3].
The literature seems to indicate that I can likely force recreate the RAID array using –create and –assume-clean, but I'm worried it will aggravate my data.
My current thoughts are to recreate the array in a degraded state using the drives with the highest matching events (sdc1, sdd1 and sde1, below). Is there a better solution?
megatron ~ # mdadm -E /dev/sd[bcdefghijklmnop]1 | egrep 'Event|/dev'
/dev/sdb1:
Events : 494734
/dev/sdc1:
Events : 502154
/dev/sdd1:
Events : 502154
/dev/sde1:
Events : 502154
/dev/sdf1:
Events : 494756
[1] This is the output of mdstat, after a reboot:
megatron ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : inactive sde1[3](S) sdf1[2](S) sdc1[1](S) sdd1[4](S) sdb1[0](S)
7814052444 blocks super 1.2
unused devices: <none>
[2] If I stop the array and reassemble it, I get the following:
megatron ~ # mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sde2
mdadm: /dev/sde2 has wrong uuid.
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sde has wrong uuid.
mdadm: no RAID superblock on /dev/sdf
mdadm: /dev/sdf has wrong uuid.
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdd has wrong uuid.
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has wrong uuid.
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda3: Device or resource busy
mdadm: /dev/sda3 has wrong uuid.
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as -1
mdadm: added /dev/sdc1 to /dev/md0 as -1
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives and 2 spares - not enough to start the array.
[3] Lastly, the output of mdadm -E /dev/sd[bcdefghijklmnop]1 yields:
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 09ce480d:aad5f0ac:4cfbc777:7303d4a7
Update Time : Tue Jul 9 20:16:22 2013
Checksum : 8a17d4c9 - correct
Events : 494734
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d89673c0:5efc51a7:1589bbc2:d7002e7d
Update Time : Wed Jul 10 00:14:04 2013
Checksum : 57efa473 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 08542b46:f4cfa093:31650e72:dd781e74
Update Time : Wed Jul 10 00:14:04 2013
Checksum : aa71a2c3 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8c655b9d:4630e93b:32eb61b8:f7a5f513
Update Time : Wed Jul 10 00:14:04 2013
Checksum : 55dcae05 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3b54d74e:a1065dd6:d3e836ca:b340b56d
Update Time : Tue Jul 9 20:17:04 2013
Checksum : d63b6fa - correct
Events : 494756
Layout : near=2
Chunk Size : 512K
Device Role : spare
Array State : .AAA ('A' == active, '.' == missing)
Best Answer
Wrong two drives failed in RAID10 4+1 configuration will break the array, yes :(
It looks lucky for you that three of them ARE in sync (as you say, with 502154), so yes, you should try to reassemble three of them. It looks pretty hopefull.
Of course, if you had 4 extra drives you should make copies before trying in case it does not go so well...