Server freezes without kernel panic

The error seems to be from an 'invalid opcode' of 0000 which means the kernel tried to jump to execute memory that was zero'd. You kernel is tainted with 'D' so this means you've had oopses recently, you should look for these in /var/log/messages and elsewhere as they may be indicators the actual root cause.

I would say that you are likely right about it happening on high IO as it happened while the rsync process was on the CPU. That doesn't make it certain to be IO related, but it does raise eyebrows.

Given the age of the machine you could be facing a hardware problem. Have you tried running memtest on the machine? Or running an IO benchmark (like bonnie++) to see if that causes the crashes to happen?

1. Wait for the RAID resync to complete. Since it's started, if it can run to completion, that'll be best.
2. Reboot the system in single-user mode, unmount the /home partition (if it's mounted) and do an e2fsck -f /dev/md2, to ensure the file system is self-consistent.
3. If, after that, this happens again, it's likely indicative either of bad hardware or of a genuine kernel bug. If you can't find any evidence of the former, make sure you're running the very latest C5 kernel with a yum update, and if so, log a bug with the CentOS project via their bug tracker (after searching to ensure this is not a known issue).

Edit: I wouldn't regress the kernel without compelling evidence that there's some known issue with the current kernel. If your kernel is logging continuous, progressive FS corruption then for me that is very strongly indicative of hardware issues.

Have you run smartctl checks on the sdc and sdd discs? You say "the discs are fine", but you don't say how you know that.

If the discs really are fine, then I notice that you're using only a partition on sdc and sdd to provide the metadevice - something that's worth checking is that the partition tables don't overlap. I've known problems be caused when partitions overlapped by a few blocks, because the superblock at the bottom of one file system kept treading on the blocks at the very top of the other FS.

Edit 2: thanks for the smartctl output. Unfortunately, the "health check is passed" output is fairly meaningless, because neither disc has ever been tested ("No self-tests have been logged"). Try a smartctl -t long /dev/sdc, and when it's finished, the same for sdd, and then see what smartctl says.