Can’t get the RAID array out of degraded mode

disaster-recoveryraidraid1raid10software-raid

I've got a 4-drive RAID 10 array that has just had a drive failure. I ignorantly have never practiced how to recover from a failure (I'm a programmer, just keeping this server as a hobbyist) so I'm having to learn this all the hard way right now.

I managed, through Google and this site (thank you guys!) to figure out how to fail, remove, add and resync the new drive, but it keeps failing during the resync and simply marking the new disk as a spare.

With more Googling and some more command-line-fu, I've figured out that the remaining "good" drive actually has some bad sectors producing read errors during the sync, so mdadm is aborting and marking as spare.

I've used badblocks to confirm the presence of bad sectors (there appear to be quite a few), but I have no idea if those sectors are actually in use or not (so far, I haven't noticed any corrupt data). I've also read that fsck can potentially repair this data, but I've also read that it has the real possibility of completely fubaring the drive, too. As such, I haven't tried it, yet.

I've tried using mdadm's --force flag to ignore these errors during re-sync, but it doesn't appear to help at all.

I have all my critical data backed up already, but there is a large amount of non-critical data that I'd really rather not lose if it can be avoided (it's all replaceable, but would take a very long time). Additionally, my critical backups are all in the cloud, so even restoring those, while easy, would be very time-consuming.

Also, if needed, I have one more unused, new replacement drive on hand.

Below is all the info about the system that I know to give. Please let me know if you need more! How do I get this array fully rebuilt?

Drive Layout

sda + sdb = RAID1A (md126)

sdc + sdd = RAID1B (md127)

md126 + md127 = RAID10 (md125)

The problem array is md126, the new unsynced drive is sdb, and the problem drive is sda

root@vault:~# cat /proc/mdstat

Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]
md125 : active raid0 md126p1[1] md127p1[0]
      5860528128 blocks super 1.2 512k chunks

md126 : active raid1 sda1[1] sdb1[2](S)
      2930265390 blocks super 1.2 [2/1] [U_]

md127 : active raid1 sdc1[1] sdd1[0]
      2930265390 blocks super 1.2 [2/2] [UU]

unused devices: <none>

root@vault:~# parted -l

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid

root@vault:~# sudo mdadm --detail /dev/md126

/dev/md126:
        Version : 1.2
  Creation Time : Thu Nov 29 19:09:32 2012
     Raid Level : raid1
     Array Size : 2930265390 (2794.52 GiB 3000.59 GB)
  Used Dev Size : 2930265390 (2794.52 GiB 3000.59 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jun  2 11:53:44 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           Name : :RAID1A
           UUID : 49293460:3199d164:65a039d6:a212a25e
         Events : 5200173

    Number   Major   Minor   RaidDevice State
       1       8        1        0      active sync   /dev/sda1
       2       0        0        2      removed

       2       8       17        -      spare   /dev/sdb1

Edit: Here are the contents of the kernel log during the failed recovery process.

root@vault:~# mdadm --assemble --update=resync --force /dev/md126 /dev/sda1 /dev/sdb1

root@vault:~# tail -f /var/log/kern.log

Jun  5 12:37:57 vault kernel: [151562.172914] RAID1 conf printout:
Jun  5 12:37:57 vault kernel: [151562.172917]  --- wd:1 rd:2
Jun  5 12:37:57 vault kernel: [151562.172919]  disk 0, wo:0, o:1, dev:sda1
Jun  5 12:37:57 vault kernel: [151562.172921]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 12:37:57 vault kernel: [151562.173858] md: recovery of RAID array md126
Jun  5 12:37:57 vault kernel: [151562.173861] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  5 12:37:57 vault kernel: [151562.173863] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Jun  5 12:37:57 vault kernel: [151562.173865] md: using 128k window, over a total of 2930265390k.
Jun  5 12:37:57 vault kernel: [151562.248457]  md126: p1
Jun  5 12:37:58 vault kernel: [151562.376906] md: bind<md126p1>
Jun  5 13:21:52 vault kernel: [154196.675777] ata3.00: exception Emask 0x0 SAct 0xffe00 SErr 0x0 action 0x0
Jun  5 13:21:52 vault kernel: [154196.675782] ata3.00: irq_stat 0x40000008
Jun  5 13:21:52 vault kernel: [154196.675785] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:52 vault kernel: [154196.675791] ata3.00: cmd 60/00:48:a2:a4:e0/05:00:38:00:00/40 tag 9 ncq 655360 in
Jun  5 13:21:52 vault kernel: [154196.675791]          res 41/40:00:90:a7:e0/00:05:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:52 vault kernel: [154196.675794] ata3.00: status: { DRDY ERR }
Jun  5 13:21:52 vault kernel: [154196.675797] ata3.00: error: { UNC }
Jun  5 13:21:52 vault kernel: [154196.695048] ata3.00: configured for UDMA/133
Jun  5 13:21:52 vault kernel: [154196.695077] sd 2:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:52 vault kernel: [154196.695081] sd 2:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:52 vault kernel: [154196.695085] sd 2:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:52 vault kernel: [154196.695090] sd 2:0:0:0: [sda] tag#9 CDB: Read(16) 88 00 00 00 00 00 38 e0 a4 a2 00 00 05 00 00 00
Jun  5 13:21:52 vault kernel: [154196.695092] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:52 vault kernel: [154196.695111] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.675248] ata3.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0
Jun  5 13:21:55 vault kernel: [154199.675252] ata3.00: irq_stat 0x40000008
Jun  5 13:21:55 vault kernel: [154199.675255] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:55 vault kernel: [154199.675261] ata3.00: cmd 60/08:c0:8a:a7:e0/00:00:38:00:00/40 tag 24 ncq 4096 in
Jun  5 13:21:55 vault kernel: [154199.675261]          res 41/40:08:90:a7:e0/00:00:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:55 vault kernel: [154199.675264] ata3.00: status: { DRDY ERR }
Jun  5 13:21:55 vault kernel: [154199.675266] ata3.00: error: { UNC }
Jun  5 13:21:55 vault kernel: [154199.676454] ata3.00: configured for UDMA/133
Jun  5 13:21:55 vault kernel: [154199.676463] sd 2:0:0:0: [sda] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:55 vault kernel: [154199.676467] sd 2:0:0:0: [sda] tag#24 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:55 vault kernel: [154199.676471] sd 2:0:0:0: [sda] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:55 vault kernel: [154199.676474] sd 2:0:0:0: [sda] tag#24 CDB: Read(16) 88 00 00 00 00 00 38 e0 a7 8a 00 00 00 08 00 00
Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.
Jun  5 13:21:55 vault kernel: [154199.855992] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.855995]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.855998]  disk 0, wo:0, o:1, dev:sda1
Jun  5 13:21:55 vault kernel: [154199.856000]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 13:21:55 vault kernel: [154199.872013] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.872016]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.872018]  disk 0, wo:0, o:1, dev:sda1

Best Answer

The critical lines are

Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.

To recover md126 the kernel needs to copy sda1 to sdb1 - but it's having a read error on sda, which is the only surviving half of that mirror. This array is toast. It's time to shotgun /dev/sda and restore from backups (or, absent backups, save what you can from the existing array before shotgunning it).

Edit: If you're trying to get data off the failing drive, tools like safecopy may come in useful (disclaimer: I have no connection with the authors or the project).

Related Topic