The simple answer is that to mirror something takes almost no processing power - it just writes to the disk a second time. For RAID-Z2, you have to compute an entirely new parity block, which although small CAN bog down the CPU when you have to write large amounts of data quickly.
Mirroring is always the preferred solution for high-speed data, if it's just bulk-storage without fast write speeds, RAID-Z2 is a good alternative that does allow any two drives to die as you allude to.
The other advantage is that mirrored pools can be expanded with more mirrored devices - while a RAID-Z2 can not be expanded - though more RAID-Z2 storage can be added to the pool, it will be two RAID-Z2 storage pools concatenated (in effect) rather than equally split between all the storage and striped.
It's been mentioned that RAID is not a backup. VERY TRUE. Keep that in mind.
You're using terabyte sized disks, which increases the chances of an unrecoverable read error, which is a MAJOR PAIN IN THE @#$. Raid 5 is almost unusable as disks get larger; you could have one of the three disks fail completely, you replace it, and that's when you discover that one of the "good" disks has a spot that can't be read from, so you end up having to completely rebuild from backup. We had that happen with a hardware-based RAID (PERC controller).
Your RAID level depends on how you're using the server. I like 1 for most of my purposes (mirroring). It has very good read times because it can spread read commands across drives, but writes can suffer somewhat. How affected it is depends on what you're using for the controller and drive speed. Go to Wikipedia and search for RAID to get a rundown of RAID levels; no one can really tell you what to definitively use without knowing your workload, the server's usage, etc.
Do not use rsync for a backup on the same computer. If your controller is fried or something goes weird on the computer itself (or the machine is damaged in flooding, fire, electrical surge) you risk the backup getting toasted too. Backup means being able to rebuild your data on new hardware if need be after a catastrophic failure.
If you're referring to a hardware RAID controller built into the motherboard-don't. don't don't don't. Motherboard RAID is cheap, crappy, and cheap, and worse than any software-implemented RAID. If you want to go through the trouble of building a production system with RAID, use either the built-in Linux/BSD software RAID or get a good RAID card like one from 3Ware. Personally for a server, I'd get a hardware card and search the specs for features like hot swap capability and lighted alarms to indicate WHICH DRIVES have failed. There's nothing wrong with performance or ability of software RAID, and it's very reliable, but there are many questions about "I have a drive that failed and don't know which one it is", and if you screw it up you can break your data set or erase the wrong data. System administration is supposed to have some element of making your life easier (hee hee!) and puzzling which drive is which cable is which mountpoint is not fun. The hardware cards are $$ but often save you much frustration when trying to puzzle out which is in need of replacement.
Don't skimp on hard drive speed. Faster, the better, especially if this is a heavy usage server. Today's gig lans can easily make the hard disk a bottleneck now for big transfers or heavy sharing.
Make sure you have a way to monitor the RAID, and make it a point to periodically check the status of your drives.
Get a good backup system in. Any fileserver should have a good second-machine backup, whether to tape or disk. If your server blows up tomorrow you should be able to get parts in and start restoring everything from scratch if need be, unless the business issuing the paychecks can survive without their server, in which case I don't know why you'd be worried about RAID.
Hope this helps!
Best Answer
Don't know about the speed, but here is what I believe running ZFS on RAID would means:
You lose the benefits of atomic writes because now the RAID controller has the last say on when a write happens to the disk. Which means you rely on the RAID controllers NVRAM.
ZFS also may get lied to if the data was written improperly. ZFS would have to take the RAID controllers word for it.
You would also lose repairing files because, from ZFS's point of view, you have a single disk, if the data's integrity is bad, ZFS would have no way to repair it because there is no second copy. (assuming you don't set copies=2)
If the RAID drives fall out of sync the RAID drive may take some time to sync depending on the journaling ability. ZFS will resilver the data it finds bad and at least some OSs may run a resilver periodically to ensure the integrity. Again because the RAID will only display one drive to ZFS, ZFS can't help with the repair/rebuild.
You would be able to expand the RAID (if the RAID has the capability) and maybe rebuild the ZFS data across more drives. (For me not a big plus considering the negatives so far.)
Of course all the snapshot functionality of ZFS would be unaffected (assuming the data doesn't silently get corrupted).
Hardware RAID would almost negate any advantages that ZFS would have. Personally, I wouldn't recommend using anything under ZFS, I would run ZFS on bare metal.
However, if there is an advantage I hadn't considered, I'm open to hearing it.