Centos – md/raid:md2: cannot start dirty degraded array, kernel panic

centoskernel-panic

After having made use of a remote power switch, my server did not come back online.
When I went to the datacenter and reboot the computer on the spot I see the server booting (I see the centos progress bar with running almost all the way to the end) and eventually giving the following messages:

md/raid:md2: cannot start dirty degraded array.
md/raid:md2: failed to run raid set.
md: pers->run() failed ...
md/raid:md2: cannot start dirty degraded array.
md/raid:md2: failed to run raid set.
md: pers->run() failed ...



Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init not tainted 2.6.32-279.1.1.el6.i686 #1
Call Trace:
 [<c083bfbc>] ? panic+0x68/0x11c
 [<c045a501>] ? do_exit+0x741/0x750
 [<c045a54c>] ? do_group_exit+0x3c/0xa0
 [<c045a5c1>] ? sys_exit_group+0x11/0x20
 [<c083eba4>] ? syscall_call+0x7/0xb
 [<c083007b>] ? cmos_wake_setup+0x62/0x112

The server runs CentOS and has software raid, and I don't have backups of the raid settings. The only backup I have is of /home and the database dumps. (Glad to at least have those though.)

Since the server is an old Dell PowerEdge 1750 with no CD-ROM drive, I have no way of booting the machine from a boot disk. I also remember in the past that the server also wouldn't boot from a bootable USB disk. So the only way I know how to boot the server is to go to the datacenter, pick up the server and take it to the office. Screw open the server. Attach a cdrom drive to an IDE slot on the motherboard. And then boot it. I am hoping you guys could help me avoid this.

I have looked a bit through the boot options and I found the following boot options. When CentOS is about to boot and interrupt the boot-countdown:

CentOS (2.6.32-279.1.1.el63.i686)
CentOS Linux (2.6.32-71.29.1.el6.i686)
centos (2.6.32-71.el6.i686)

I think the first configuration is the default one, because choosing that gets me to the above mentioned kernel panic. The other ones end with something like "Sleeping forever".

I can press 'e' to edit boot commands, press 'a' to modify kernel arguments and press 'c' for grub command line.

The command line gives a grub> prompt. But I have no idea how to get the system to boot without (trying to) access the dirty partitions.

What I want to do is off course:
– boot the machine
– check hard drive for errors
– mark the drive as clean

Best Answer

From the grub screen, edit your boot commands and add emergency to the end of the boot command line options. Then boot it up. This isn't guaranteed to work (if md2 is your root filesystem it will fail). If this fails to get you a shell, you will have to go find a CD-ROM drive.

Once you have a shell, you can run mdadm to attempt to recover your RAID array.

Find out what devices are supposed to be part of it:

mdadm -D /dev/md2

You'll see a listing of devices. If some are marked as removed or failed then you'll have to deal with the failed disks first.

After that, reassemble your RAID array:

mdadm --assemble --force /dev/md2 /dev/**** /dev/**** /dev/**** ...

(* listing each of the devices which are supposed to be in the array from the previous output.)

Related Topic