In any of the RAID levels that use striping, increasing the number of physical disks usually increases performance, but also increase the chance of any one disk in the set failing. I have this idea that I shouldn't use more than about 6-8 disks in a given RAID set but that's more just passed down knowledge and not hard fact from experience. Can anyone give me good rules with reasons behind them for the max number of disks in a set?
Maximum Number of Disks in a RAID Set – Guidelines
raid
Related Solutions
My preferred layout would be:
2 disks for the OS in RAID1 (protected against a HDD failure)
2n disks for the RAID10 (protected against a HDD failure)
rest of the disks for RAID0 (no redundancy, maximum speed)
If you really insist on having a single system disk, it doesn't change the layout.
On (each of) the system disks I'd have a separate partition for /boot filesystem. The rest of the disk(s) I'd made into an LVM physical volume. I like to have OS filesystems that can fill up on separate logical volumes (/tmp, /var, /opt if something writes logs in there). So that would result in /boot (mirrored, no LVM) and /, /tmp, var and possibly /opt filesystems, each on a separate logical volume on a mirrored disk.
On each of the other disks I'd create a single partition of the type Linux raid and create appropriate RAID arrays (one RAID 10, one RAID 0). On each of the arrays I'd create a single partition of the type Linux LVM and make two separate volume groups, one for redundant and one for non-redundant data. Then for each file system you plan to make I'd make a logical volume.
In each volume group I would recommend to leave some space unused, so that you can do an LVM snapshot and do fsck of a file system without bringing the server down. I would also disable automatic fsck of all filesystems (tune2fs -i 0 -c 0 /device/name).
Rationale
1) Mirroring of the OS disks.
Failure of the system HDD brings down the whole machine. Your data is protected, but your production stops until you can bring a replacement disk and reinstall / restore the OS. In a production environment it is usually cheaper to have one more disk installed.
2) Partitioning disks for RAID arrays.
All the servers I use have partition tables. You may use just whole disks as RAID / LVM volumes, but then you end up with some machines that have partition tables (stuff on /dev/sdX1) and some, which don't (stuff on /dev/sdX). In case of a failure and need for recovery under stress I like to have one variable less in the environment.
3) LVM on the RAID arrays
LVM gives two advantages: easy changes of filesystem sizes and ability to fsck filesystems without bringing the whole server down. Silent data corruption is possible and happens. Checking for it may save you a lot of excitement.
4) tune2fs -i 0 -c 0
Having a surprise fsck of a large filesystem after a reboot is a time-consuming and nerve-whacking affair. Disable it and do regular fscks of LVM snapshots of filesystems.
A question: /opt/backup is where you plan to keep backups of your production environment?
Don't.
Have the backups somewhere else from the machine. A malicious program, a mis-spelled command (e.g. rm -rf / tmp/unimportant/file) or some water spilled in / flooding the wrong place will leave you without your system and with no backups. If all else fails, have two external USB disks for backups, still better than a partition inside the same box.
In theory yes, more drives in a raid0 would lead to higher performance because the load is shared over more drives. However in practice you would be limited by the the bandwidth of the raid controller, the CPU and memory performance and similar. The performance increase would not be linear, that is 4 disks is not exactly twice as fast as 2 disks.
In any reasonably modern system with a raid controller, or even using a software raid with linux' mdadm, using 8 drives will be faster than using 2 and you should not be held back by the rest of the system's performance. The CPU, raid and/or disk controller, memory, it all should be able to handle it. You may see increased use of system resources the more drives you add. Especially if you use the onboard SATA controller in a softraid combination. But nothing that would really hinder overall usability. If using linux you may want to use a kernel that has been configured without "preempt" in order for server oriented tasks to get preference over user responsiveness.
https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO
Of course the more drives you add, the higher the chance one of them fails and your whole raid is destroyed. I would expect a raid0 of 8 drives to not last more than a year or two, if you're lucky. A raid0 of 16 drives would be asking for trouble and then I'd consider a raid10, it would still be fast enough and you have less to worry about.
As for how many drives would max out a system's resources I wouldn't know unless I had detailed system specs. I think you'd be limited more by the failure rate, if you go over about 16 disks (I rather not like to think about it).
Naturally you'd only use the raid0 for data that can be lost any time without problems. It would work great for things such as a build server, or scratch space for large scientific computations. In fact those scenarios is what I often used a raid0 for and it is a great way to squeeze a bit more life out of a bunch of older, lower capacity and inexpensive disks that would otherwise have been collecting dust. You can even mix sizes, at least with mdadm.
If using mdadm it may be worth considering to just use a raid10 as in certain configurations it can get near the performance of a raid0, that is read performance of a raid0 and already improved write performance over other raid levels (except raid0). You would get better redundancy than other raid levels, with only a slight speed penalty compared to a raid0. That would be the best of both worlds, you don't find that often.
https://en.wikipedia.org/wiki/RAID#Non-standard_levels
Linux MD RAID 10 provides a general RAID driver that in its "near" layout defaults to a standard RAID 1 with two drives, and a standard RAID 1+0 with four drives; though, it can include any number of drives, including odd numbers. With its "far" layout, MD RAID 10 can run both striped and mirrored, even with only two drives in f2 layout; this runs mirroring with striped reads, giving the read performance of RAID 0. Regular RAID 1, as provided by Linux software RAID, does not stripe reads, but can perform reads in parallel.
As suggested in the comments, mixing sizes with mdadm will not give a speed increase if you utilise all disk space as opposed to letting the smallest disk define the size of the array.
Also seek time will not improve in a raid0 and can even become a bit slower. For an SSD based raid0 the seek time would be so small (between 0.08 and 0.16 ms https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#cite_note-HP_SSD-6) it wouldn't matter much I expect.
Best Answer
The recommended maximum number of disks in a RAID system varies a lot. It depends on a variety of things:
For SATA-based RAID, you don't want to have more than about 6.5TB of raw disk if you're using RAID5. Go past than and RAID6 is a much better idea. This is due to the non-recoverable read error rate. If the size of the array is too large, the chances of a non-recoverable read error occurring during the array rebuild after a loss get higher and higher. If that happens, it's very bad. Having RAID6 greatly reduces this exposure. However, SATA drives have been improving in quality lately, so this may not hold true for much longer.
The number of spindles in an array doesn't really worry me over much, as it's pretty simple to get to 6.5TB with 500GB drives on U320. If doing that, it would be a good idea to put half of the drives on one channel and half on the other just to reduce I/O contention on the bus side. SATA-2 speeds are such that even just two disks transferring at max-rate can saturate a bus/channel.
SAS disks have a lower MTBF rate than SATA (again, this is beginning to change) so the rules are less firm there.
There are FC arrays that use SATA drives internally. The RAID controllers there are very sophisticated, which muddies the rules of thumb. For instance, the HP EVA line of arrays groups disks into 'disk groups' on which LUNs are laid out. The controllers purposefully place blocks for the LUNs in non-sequential locations, and perform load-leveling on the blocks behind the scenes to minimize hot-spotting. Which is a long way of saying that they do a lot of the heavy lifting for you with regards to multiple channel I/O, spindles involved in a LUN, and dealing with redundancy.
Summing up, failure rates for disks doesn't drive the rules for how many spindles are in a RAID group, performance does. For the most part.