I have a weird issue with a server I am setting up. It's for a filesharing-type website, so fast IO and plenty of capacity are the requirements. OS is CentOS 6.4 64-bit
The server in question is a HP DL360p, with 18x drive bays populated with 2TB SAS drives in RAID50
There's also a HP StorageWorks SAS expansion bay with a further 12x2TB's, also in RAID50
RAID was configured using the server's BIOS configuration utility, the controllers used are pretty good ones with battery backup and 2GB FWBC.
Now, originally we set these up as separate volumes, but due to the specifics of our software, it would work out much better to have a single, large volume.
So, I set up an LVM volume combining these two volumes, then formatted the logical volume using XFS
The problem is, the resulting speed is disappointing. Running hdparm -tT gives a best read speed of 300MB/s
So I did a few tests and got this:
no LVM, XFS on both: Both volumes get around 700MB/s read speeds
with LVM, but volume not mounted: 1000-1100MB/s
with LVM in striped mode, volume not mounted: 1100-1300MB/s
So somehow XFS seems to be restricting the speed… I tried some more advanced options when formatting and mounting, such as -l internal, enabling lazy-count, nobarrier, but this yielded no improvement.
The only thing I found that may be a problem, the strip size of the RAID volumes did not match (one was set at 512KB and the other as 256KB), so I am re-configuring them to match, which will take a few hours more. I also reformatted the volume with su=512k,sw=28 (sw=28 is just a guess, as there are 28 active HDDs altogether… or should I set this to 2 for the RAID volumes?)
I'm tempted to just wipe out the whole thing and try ZFS, it seems promising, but I think configuring it would be far beyond my skill level…
So, if anyone has any experience or advice on this, it would be much appreciated!
Best Answer
What are your application's read/write throughput and IOPS requirements? Storage performance is not always about the array throughput or raw bandwidth. Sequential reads/writes are only a faction of the I/O activity.
A more accurate test would be a bonnie++ or iozone run against a mounted filesystem... Or even running your application and metering the real workload.
If I were you, I'd dump the internal and external controllers and consolidate to an HP Smart Array P822 controller - Part #615418-B21.
This would allow you to address your internal disks and the external enclosure in one array. The P822 also has the Smart Array Advanced feature set (SAAP) enabled by default. At that point, you can carve the array up properly with the right RAID level (probably RAID 1+0) and the ability to assign one or more global hot-spares to the setup. The controller will also leverage the dual-paths to your external storage. You could also arrange things to stripe or mirror your drive pairs between internal and external storage. A lot of flexibility.
The biggest advantage of this setup, though, is the included HP SmartCache SSD caching feature. This is similar to LSI's Cachecade. Armed with an SSD or two, you can keep hot data on lower-latency SSDs versus having to go to spinning disk.
That's just the hardware side, though...
For XFS volumes and especially on HP gear, I don't use LVM (some people do, though). With HP controllers, the block device presentation is abstracted, so I use a pretty basic filesystem formatting string:
The fstab has a couple of mount options:
But with RHEL6, there are also some scheduling and performance tuning features you should take into account. The main one is the tuned framework.
This will set the I/O elevator, disable write barriers and set a few other performance-minded options on-the-fly according to the schedule below.
ZFS on this setup won't help you. You'd have to dump the Smart Array controllers entirely and go to SAS HBAs or lose all ZFS RAID functionality and use one large block device (which would benefit from the Smart Array P822 I proposed above). Both would require some form of write cache (ZIL) and I don't think that it solves the manageability problem. ZFS requires way more planning up-front.