Disks for a small esx-host: ssd raid5 or hdd raid10?

You cannot mix drive types in the same array group on PERC/LSI/HP hardware RAID controllers.

This also applies to mixing SATA, SAS and SSD.

PERC manual:

"Mixing SAS and SATA drives within a virtual disk is not supported. Also, mixing hard drives and SSDs within a virtual disk is not supported."

One aspect of my job is designing and building large-scale storage systems (often known as "SANs", or "Storage Area Networks"). Typically, we use a tiered approach with SSD's and HDD's combined.

That said, each one has specific benefits.

1. SSD's almost always have a higher Cost-per-Byte. I can get 10k SAS 4kn HDD's with a cost-per-gigabyte of $0.068/GB USD. That means for roughly $280 I can get a 4TB drive. SSD's on the other hand typically have a cost-per-gigabyte in the 10's and 20's of cents, even as high as dollars-per-gigabyte.

2. When dealing with RAID, speed becomes less important, and instead size and reliability matter much more. I can build a 12TB N+2 RAID system with HDD's far cheaper than SSD's. This is mostly due to point 1.

3. When dealt with properly, HDD's are extremely cheap to replace and maintain. Because the cost-per-byte is lower, replacing an HDD with another due to failure is cheaper. And, because HDD failures are typically related to time vs. data-written, replacing it doesn't automatically start using up TBW when it rebuilds the RAID array. (Granted, TBW percentage used for a rebuild is tiny overall, but the point stands.)

4. The SSD market is relatively complex. There are four (current, at the time of this writing) major types of SSD's, rated from highest number of total writes supported to lowest: SLC, MLC, TLC, QLC. The SLC typically supports the largest numbers of total writes (the major limiting factor of SSD lifetimes), whereas the QLC typically supports the lowest numbers of total writes.

That said, the most successful storage systems I've seen are tiered with both drives in use. Personally, all the storage systems I recommend to clients generally follow the following tiers:

1. Tier 1 is typically a (or several) RAID 10 SSD-only tier. Data is always written to Tier 1.
2. Tier 2 is typically a (or several) RAID 50 or 5 SSD-only tier. Data is aged out of Tier 1 to Tier 2.
3. Tier 3 is typically a (or several) RAID 10 HDD-only tier. Data is aged out of Tier 2 to Tier 3.
4. Tier 4 is typically several groups of RAID 6 HDD-only tiers. Data is aged out of Tier 3 to Tier 4. We make the RAID 6 groups as small as possible, so that there is a maximal support of drive-failure.

Read/Write performance drops as you increase tiers, data will propagate down to a tier where most of the data shares the same access-/modification-frequency. (That is, the more frequently data is read/written, the higher the tier it resides on.)

Sprinkle some well-designed fibre-channel in there, and you can actually build a SAN that has a higher throughput than on-board drives would.

Now, to some specific items you mention:

Your SSD Questions

How SSD exactly works, when life cycle for one cell is out, what then? Disk is reduced by only this cell and works normally? Or what happened then?

• Both drive-types are typically designed with a number of "spare" cells. That is, they have "extra" space on them you cannot access that supports failing-to if a cell dies. (IIRC it's like 7-10%.) This means if a single "cell" (sector on HDD) dies, a "spare" is used. You can check the status of this via the S.M.A.R.T. diagnostics utility on both drives.

What is best solution (filesystem) to write? I think ext4 is good, because it saves to cells consecutively?

• For SSD's this is entirely irrelevant. Cell-positioning does not matter, as access time is typically linear.

Your HDD Questions

In case of mechanical fault, no way to repair it (is it right)?

• Partially incorrect. HDD's are actually easier to recover data from in most failure situations. (Note: I said easier, not easy.) There is specialized equipment required, but success-rates here seem pretty high. The platters can often be read out of the HDD itself by special equipment, which allows data-recovery if the drive is dead.

Slowest, but I think speed is not so important, because speed of HDD is absolutely sufficient for server using?

• Typically, when using RAID, single-drive speed becomes less a factor as you can use speed-pairing RAID setups that allow you to increase the overall speed. (RAID 0, 5, 6 are frequently used, often in tandem.) For a database with high IO's, HDD's are typically not sufficient unless designed very deliberately. You would want SLC write-intensive grade SSD's for database-grade IO.