I had a 3 disk mdadm raid5 volume (made of /dev/sd[abc]) that was in the middle of a reshape, going to 5 disks (adding /dev/sd[df]), when the power to the server failed. this is on ubuntu 8.10 intrepid 64bit server (kernel 2.6.27), mdadm version 2.6.4, built from the source. The ups monitor daemon was nice enough to execute the following commands at that time
mdadm --stop /dev/md0
shutdown -P now
when i got the box back up and running, i used the following command to bring the raid back up, since it wouldnt come up by itself.
mdadm --assemble /dev/md0 /dev/sd[abcdf]
it came back up and restarted along its reshape process, but with only 4 discs. /dev/sdf had no superblock, so it wouldnt come along for the ride. i tried adding it back in:
mdadm --add /dev/md0 /dev/sdf
but that just added it as a hot spare, not as the missing 5th member of the raid. so i removed it with –remove, since a hot spare at this point in the process is kinda useless. to make matters worse, this added a superblock to the disk that identifies it as a hotspare now. easy enough to get rid of with –zero-superblock though.
what i want to know is, how do i massage this disk to get it back into the array in its proper place? i have no reason to believe that the data is bad, just the superblock. or is it not even appropriate at this point to try to re-add it now that the reshape has progressed without it?
contents of /proc/mdstat:
root@FS1:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdd[4] sdc[2] sdb[1]
1953524736 blocks super 0.91 level 5, 512k chunk, algorithm 2 [5/4] [UUU_U]
[======>..............] reshape = 33.4% (326807040/976762368) finish=1175.3min speed=9216K/sec
unused devices: <none>
contents of /etc/mdadm/mdadm.conf
root@FS1:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72
# This file was auto-generated on Wed, 15 Jul 2009 09:09:57 -0400
# by mkconf $Id$
output of mdadm –examine –scan
root@FS1:~# mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=5 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72
spares=1
output of mdadm -D
root@FS1:~# mdadm -D /dev/md0
/dev/md0:
Version : 00.91.03
Creation Time : Sun Jul 12 15:57:27 2009
Raid Level : raid5
Array Size : 1953524736 (1863.03 GiB 2000.41 GB)
Used Dev Size : 976762368 (931.51 GiB 1000.20 GB)
Raid Devices : 5
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Jul 16 23:34:21 2009
State : clean, degraded, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Reshape Status : 33% complete
Delta Devices : 2, (3->5)
UUID : 4128dcfb:b5e905ae:60b633ef:af4ede72 (local to host FS1)
Events : 0.216044
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 32 2 active sync /dev/sdc
3 0 0 3 removed
4 8 48 4 active sync /dev/sdd
Best Answer
First off, let the md0 device finish digesting the existing setup. Don't monkey with it at this moment, as it's busy trying to follow its one and only imperative - get the array back into a clean state!
Once things have stabilized, proceed to remove the "old" entry for the fifth drive. Then proceed to add the drive into the array and reshape it as you initially did, although this time you'll start with 4 disks instead of 3.