Debian software RAID1 is not working when I remove hard drive

debiangrubraidsoftware-raid

I have been trying all weekend to get RAID working on an old server computer with the Intel ICH7 / ICH7-R SATA-II controller running dual hard-drive with RAID 1.

After giving up in hardware raid, I have moved on to software.

Currently, the system will boot fine off both drives, and will boot fine off sdb, but when I try to boot sda, I get a blinking cursor on a black screen.

By this, I mean physically removing drives. I remove one drive, boot and it works. Replace that drive, and remove the other, boot and it does not work.

My guess is that I did not install GRUB properly on sda.

I remove the sdb hard drive, and boot the installation disk into recovery mode. I then mount RAID volume, and go into shell.

First, I will try what this tutorial that I followed for software RAID told me to do:

# grub
grub> device (hd0) /dev/sda

grub> root (hd0,0)
 Filesytem type is ext2fs, partition type 0xfd

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists ... no
 Checking if "/grub/stage1" exists ... no

Error 2: Bad file or directory type

So I try something different:

When Typing:

grub-install /dev/sda

I get

Searching for the GRUB installation directory ... found: /boot/grub
The file /boot/grub/stage1 not read correctly.

I am using ext4 partitions.

What shall I try next?

Edit:

Here is the output of fdisk -l

root@debian:~# fdisk -l

    Disk /dev/sda: 153.4 GiB, 164696555520 bytes, 321672960 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x8225e6c2

    Device     Boot   Start       End   Sectors   Size Id Type
    /dev/sda1  *       2048    194559    192512    94M fd Linux raid autodetect
    /dev/sda2        194560   4194303   3999744   1.9G fd Linux raid autodetect
    /dev/sda3       4194304 321671167 317476864 151.4G fd Linux raid autodetect

    Disk /dev/sdb: 153.4 GiB, 164696555520 bytes, 321672960 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0x3cfa42ad

    Device     Boot   Start       End   Sectors   Size Id Type
    /dev/sdb1  *       2048    194559    192512    94M fd Linux raid autodetect
    /dev/sdb2        194560   4194303   3999744   1.9G fd Linux raid autodetect
    /dev/sdb3       4194304 321671167 317476864 151.4G fd Linux raid autodetect

    Disk /dev/md2: 151.3 GiB, 162413936640 bytes, 317214720 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk /dev/md0: 93.9 MiB, 98435072 bytes, 192256 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk /dev/md1: 1.9 GiB, 2046820352 bytes, 3997696 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes

Here is the output of mdadm --detail --scan

ARRAY /dev/md/2 metadata=1.2 name=repeater:2 UUID=cd8443a8:ca0b3a29:05496e49:b063704f
ARRAY /dev/md/0 metadata=1.2 name=repeater:0 UUID=c8a204e2:3e5a5e2c:50a136c7:d43777a7
ARRAY /dev/md/1 metadata=1.2 name=repeater:1 UUID=25bebb1e:c7f3245d:128bee5d:d58d9100

Here is the output of lsblk

NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
fd0       2:0    1     4K  0 disk  
sda       8:0    0 153.4G  0 disk  
├─sda1    8:1    0    94M  0 part  
├─sda2    8:2    0   1.9G  0 part  
└─sda3    8:3    0 151.4G  0 part  
  └─md2   9:2    0 151.3G  0 raid1 /
sdb       8:16   0 153.4G  0 disk  
├─sdb1    8:17   0    94M  0 part  
│ └─md0   9:0    0  93.9M  0 raid1 /boot
├─sdb2    8:18   0   1.9G  0 part  
│ └─md1   9:1    0   1.9G  0 raid1 [SWAP]
└─sdb3    8:19   0 151.4G  0 part  
sr0      11:0    1  1024M  0 rom

EDIT 2: Here is the output of cat /proc/mdstat

root@debian:~# cat /proc/mdstat
Personalities : [raid1] 
md1 : active (auto-read-only) raid1 sdb2[1]
      1998848 blocks super 1.2 [2/1] [_U]

md0 : active raid1 sdb1[1]
      96128 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sda3[0]
      158607360 blocks super 1.2 [2/1] [U_]
      bitmap: 2/2 pages [8KB], 65536KB chunk

unused devices: <none>

Best Answer

From your lsblk output and /proc/mdstat, I see, that you have all RAIDs degraded. See [_U] or [U_]? In each RAID only one row is populated (marked with U), the other one is not (marked with _).

So, it looks my guess was correct: you have normal /boot file system only on one drive, that is now called sdb. Your sda has a partition destined for same thing, but that partition doesn't seem to contain valid and correctly populated boot file system.

If you are certain both your drives are fine, you could re-add them to raids. You need to wipe md raid superblocks from unused paritions. The mdadm tool will complain if you try to wipe labels from active drives, but it is still worth double-checking your commands. From your /proc/mdstat and fdisk output, we deduce that:

  • md0 (boot) should be on sda1 and sdb1, but only sdb1 is working.
  • md1 (swap) should be on sda2 and sdb2, but only sdb2 is working.
  • md2 (root) should be on sda3 and sdb3, but only sda3 is working - which is strange, this is on another disk.

To wipe label from sda1, sda2, sdb3 (all unused currently) you use:

mdadm --zero-superblock /dev/sda1
mdadm --zero-superblock /dev/sda2
mdadm --zero-superblock /dev/sdb3

It might complain something, you might need to add --force to the command, but I think that wouldn't be necessary. I speak again, double-check everything to wipe raid marks only on unused devices!

Then, you add these unused devices to their arrays:

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2
mdadm --add /dev/md2 /dev/sdb3

It will start resync in background. Also note, it will detect all arrays use same devices, so it will for example delay md2 resync until md1 is done. You will see resync progress in /proc/mdstat.

When resync completes, all devices in /proc/mdstat will have [UU] after them, indicating these arrays are in optimal state.

Your /boot works, so sdb1 contains correctly populated filesystem. When your /boot md0 is in optimal state, we can say, sda1 now has the same data (only superblock differs slightly, because different backing devices in same array have different row numbers). You could re-install grub by hand:

device (hd0) /dev/sda
root (hd0,0)
setup (hd0)

I strongly suggest you to set up mdadm to do array check periodically (say, once a week or once a month), to catch up drive read errors early. I've once solved a problem when both raid drives had bad blocks on them (in different places), and regular checking could avoid this problem by ejecting faulty drive early, even when bad block is in rarely used part of file system. That was tricky. To start up a check, do

echo "check" >> /sys/block/md127/md/sync_action

and look for errors in dmesg and mdmon logs.

DO NOT simply dd boot file system to other device. Two file systems with same labels and same UUIDs is not a normal state. This is permitted to have transiently during recovery or some other forensics process, but not for a normally running /boot. Ubuntu mounts partitions by UUID (check /etc/fstab), and how you will guess, which one will be mounted next time? No way to tell, this is random, but this thing is not supposed to be random.

Access to raid backing devices is somewhat blocked by md code in kernel, so while they in raid, system only sees md0, which is a single device and you will not have a problem.

To maintain non-raid /boot on both disks, you have basically need to clone filesystem each time you update boot, and then always change UUID and label. This is error-prone and inconvenient. It is normal to have /boot as RAID1.

I prefer to have metadata 1.0 for /boot, because in this case metadata is placed at the end of the device, so even in case I don't have raid support, I can simply mount file system from the partititon as if there was no raid at all. But if it work for you now, better to leave things as they are now.

Related Topic