Linux – Raid 1 Failed – Primary disk 1 dead, not booting from the mirrored one

centoslinuxraidraid1software-raid

We have a server mounted using Raid-1, using two disk of 1tb each, and configured by software on Centos.
First disk (let's call it sdb) is dead and we took it away to replace it in a near future, while we decided to boot up the server using the mirrored disk sda. The thing is the system doesn't boot. it's stay in a black screen after the bios init screen. my thoughts on this is that they didn't configure the mirrored disk to mirror the startup sector of the sdb

i used a live ubuntu cd to boot the machine, and inspect the disk. My second thought is that it lacks the MBR, since when i looked into sda i saw a partition sda1 and sda2, where the first has 10gb of space, which i couldn't access. and then the second partition sda2 which has the remaining 990gb left, which is the partition that i'd need to boot since is the one with the server files and configurations

apart of this, i saw that sdba has all the system folders, such as /boot /proc /bin … etc etcc

i've red that this MAY be solved installing the grub boot loader into the disk and then it may run. but i have fear of this if it will work, because of the first partition of 10gb already created.
Others forums said this is irrecuperable, in case this is true, is there a way where i format the sdb1 partition of 10gb and install an OS to boot up the secondary partition and all it's services ?

below here, i will put all the information that i gathered, also if you need some extra relevant information, i will gladly look for it and upload it

   root@PartedMagic># sfdisk -d /dev/sda 
//
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size= 19529728, Id=fd
/dev/sda2 : start= 19531776, size=1933991936, Id=fd
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start=        0, size=        0, Id= 0

root@PartedMagic>/media/md1/boot/grub# cat device.map
//
(hd0)   /dev/disk/by-id/ata-WDC_WD1002FAEX-007BA0_WD-WMAY00918352
(hd1)   /dev/disk/by-id/ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRC581793
(hd2)   /dev/disk/by-id/usb-Kingston_DT_101_G2_00187D0F5670EC601000BFD3-0:0

/dev/md1:
        Version : 1.2
  Creation Time : Wed Oct 30 21:06:56 2013
     Raid Level : raid1
     Array Size : 966864704 (922.07 GiB 990.07 GB)
  Used Dev Size : 966864704 (922.07 GiB 990.07 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Oct 25 17:34:47 2016
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : pbx1:1
           UUID : 87040663:7e96da33:d143c52d:63d27f4a
         Events : 6512040

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8        2        1      active sync   /dev/sda2
/dev/md0:
        Version : 1.2
  Creation Time : Wed Oct 30 21:06:51 2013
     Raid Level : raid1
     Array Size : 9756544 (9.30 GiB 9.99 GB)
  Used Dev Size : 9756544 (9.30 GiB 9.99 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Oct 25 17:36:14 2016
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : pbx1:0
           UUID : 092582ab:c3cd6b70:98b1b939:a5bb67a7
         Events : 602

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8        1        1      active sync   /dev/sda1

------------------------------
root@PartedMagic>/media/md1/boot/grub# cat device.map
//
(hd0)   /dev/disk/by-id/ata-WDC_WD1002FAEX-007BA0_WD-WMAY00918352
(hd1)   /dev/disk/by-id/ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRC581793
(hd2)   /dev/disk/by-id/usb-Kingston_DT_101_G2_00187D0F5670EC601000BFD3-0:0

This is the MDMADM.CONF file

CONFIGURACION MDADM.CONF
-------------------------------------------
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=092582ab:c3cd6b70:98b1b939:a5bb67a7 name=pbx1:0
ARRAY /dev/md/1 metadata=1.2 UUID=87040663:7e96da33:d143c52d:63d27f4a name=pbx1:1

# This file was auto-generated on Wed, 06 Nov 2013 11:10:29 -0300
# by mkconf 3.1.4-1+8efb9d1+squeeze1

This is the cfg i found inside Grub conf:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  insmod vbe
  insmod vga
  insmod video_bochs
  insmod video_cirrus
}

insmod raid
insmod mdraid
insmod part_msdos
insmod part_msdos
insmod ext2
set root='(md/1)'
search --no-floppy --fs-uuid --set 307c9805-75d2-44d9-a411-465ae556d743
if loadfont /usr/share/grub/unicode.pf2 ; then
  set gfxmode=640x480
  load_video
  insmod gfxterm
fi
terminal_output gfxterm
insmod raid
insmod mdraid
insmod part_msdos
insmod part_msdos
insmod ext2
set root='(md/1)'
search --no-floppy --fs-uuid --set 307c9805-75d2-44d9-a411-465ae556d743
set locale_dir=($root)/boot/grub/locale
set lang=es
insmod gettext
set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64' --class debian --class gnu-linux --class gnu --class os {
    insmod raid
    insmod mdraid
    insmod part_msdos
    insmod part_msdos
    insmod ext2
    set root='(md/1)'
    search --no-floppy --fs-uuid --set 307c9805-75d2-44d9-a411-465ae556d743
    echo    'Loading Linux 2.6.32-5-amd64 ...'
    linux   /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md1 ro  quiet
    echo    'Loading initial ramdisk ...'
    initrd  /boot/initrd.img-2.6.32-5-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os {
    insmod raid
    insmod mdraid
    insmod part_msdos
    insmod part_msdos
    insmod ext2
    set root='(md/1)'
    search --no-floppy --fs-uuid --set 307c9805-75d2-44d9-a411-465ae556d743
    echo    'Loading Linux 2.6.32-5-amd64 ...'
    linux   /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md1 ro single 
    echo    'Loading initial ramdisk ...'
    initrd  /boot/initrd.img-2.6.32-5-amd64
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

Any help will be appreciated, it's the first time i have to deal with an error of this nature , also one the first times approaching RAIDS.

THANKS!!

Best Answer

**Before you try anything make a clone of /dev/sda if you have a 3rd 1TB drive.

Just to be clear:

/dev/sda=Current/only drive in RAID
/dev/sdb=New drive
  • Only connect /dev/sda & boot in rescue mode with the same distro
  • chroot to sysimage and use grub-install to install it on /dev/sda. Remember you're installing it to the boot sector of your hard drive and not to a partition, so do not use a partition number like /dev/sda1
  • Remove rescue USB/CD and reboot, if it boots (yay!) shut down and add /dev/sdb and reboot. Make sure BIOS is set to boot from /dev/sda first & /dev/sdb1 second.
  • Wipe any partition from /dev/sdb1 with fdisk and mirror partitions from /dev/sda to /dev/sdb with sfdisk
  • Used mdadm to add /dev/sdb1 partition to /dev/md0
  • Dont forget to run grub-install on /dev/sdb, so you dont find yourself in this situation again :)

This link should help: https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array