Running CentOS 5.6 with mdadm RAID1, 2TB, 3 partitions (boot, swap and root). I had a kernel panic in the server, and after rebooting, the system stays up for about 5-10 minutes, the it starts printing out fsck errors to the console and the root file system is put into readonly mode.
I rebooted into single-user mode, unmounted the / partition and tried to run
e2fsck -f /dev/md2
but it hung at Stage 1 for over 8 hours without updates. I notice that the RAID is being resynched (cat /proc/mdstat) — could this be impacting the e2fsck? I ran smartctl on both disks, they both came back healthy so I don't think this is a disk problem, though I don't have any clue what triggered the RAID rebuild.
Anyway, is there a problem running e2fsck on a raid device that is being resynched? Anything I can do to get fsck out of stage 1, or get more feedback on progress? Why is the error pointing to loop0 instead of one of the actual devices?
Best Answer
The (loop0) should have given me a clue. The problem was not with any hard drive filesystem -- it was with the RAM based tmpfs. My /tmp directory was mounted using a tmpfs (/dev/shm)
The ext3 errors were occurring in the RAM based tmpfs. Once I unmounted /tmp (had to force it using umount -l /tmp), the errors went away and everything started working. The question remains why an in-memory filesystem would get ext3 errors, but at least I'm up and running.