New RAID hdparm Performance Issues on Ubuntu

hardware-raidhphp-prolianthp-smart-arrayUbuntu

I just got a HP DL180 G6 that has 25X 146GB 15K SAS drives, 36GB RAM, 2X 2.0GHz Xeon 1333Mhz FSB. For fun I configured them all in a single RAID 0 and installed Ubuntu on it to see how fast it could get with 25 drives on a HP Smart Array P410 Controller w/ 512MB RAM.

When I ran hdparm -tT /dev/mapper/concorde–vg-root I get

Timing cached reads:   5658MB in  1.99 seconds = 2834.13 MB/sec
Timing buffered disk reads: 1192 MB in  3.00 seconds = 397.13 MB/sec

When I run the same command on my other server (HP DL360 G5 – 32GB RAM – 2X 2.66GHz 667Mhz FSB) that only has 4X 15K drives I get:

Timing cached reads:   13268 MB in  1.99 seconds = 6665.18 MB/sec
Timing buffered disk reads: 712 MB in  3.00 seconds = 237.17 MB/sec

I would have expected this to run 5 times faster than the old one, not slower. The server is intended to deliver streaming media and so I need super fast access and transfer to keep up with 2 1Gb network ports I hope to max out at times along with performing it's other tasks.

I just put together a bunch of copies of a 400MB MP4 file to get 45GB to copy from one directory to another and it took 96 seconds, which just seems wrong for everything I have ever heard of the performance boost of RAID 0.

It is setup as a hardware raid, is there anything I need to do in Linux to take advantage of the extra speed that should be there? Does it matter which flavor of Linux I use? I am comfortable with CentOS and Ubuntu but could do others if needed.

Is there a different command I should use to measure performance? I tried using iotop and iostat yesterday to monitor the RAID usage and couldn't get it to report any usage while copying 2GB files over FTP, so kind of stuck trying to set a benchmark, comparing it's performance across servers, and monitoring it so I know when the hard drives are maxing out and need to be replaced with SSD.

Best Answer

Wow... there's a lot to address here.

  • Disk performance isn't just about throughput. There's the notion of IOPS and latency and service times to contend with. Most workloads are a bit random in nature, so 25 enterprise disks in an array will always trump 4 disks from an IOPS perspective.

  • hdparm is not the right tool to benchmark enterprise storage. Look into purpose-built programs like iozone and fio.

An example iozone command that could be helpful for you is (run from a large directory on the disk array you wish to test): iozone -t1 -i0 -i1 -i2 -r1m -s72g

  • The design of this server means that your disk backplane is oversubscribed. There's an expander chip on the server and those 25 disks are sharing a 4-lane 6Gbps connection to the RAID controller. That means that you have a theoretical maximum throughput of 24Gbps (or 3,000 Megabyte/second) to the array. That's a ceiling, and you won't see performance beyond that point.

  • Ubuntu is almost never the best choice when hardware drivers and support are concerned.It's not officially supported by the server. CentOS or RHEL would be a better fit for this hardware.

  • HP Smart Array controllers have the ability to carve a group of disks (an array) into multiple logical drives of varying capacity and RAID levels. The example below shows a 4-disk array carved into three logical drives. One of the logical drives is configured with a different RAID level than the others.

Smart Array P420i in Slot 0 (Embedded) (sn: 0014380296200A0)

  logicaldrive 1 (72.0 GB, RAID 1+0, OK)
  logicaldrive 2 (1024.0 GB, RAID 1+0, OK)
  logicaldrive 3 (869.1 GB, RAID 5, OK)

  physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS, 900.1 GB, OK)
  physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS, 900.1 GB, OK)

  • At no point should you use RAID 0 for a logical drive here. If you can spare the space, RAID 1+0 will perform very well with this hardware combination.

  • You have LVM in place. That's not the best approach when working with these HP Smart Array RAID controllers. It's an additional layer of abstraction, and you won't see the best performance (although it can be tuned to work well).

  • Firmware. You'll want to update the firmware of the server and related components. Functionality improves with each HP RAID controller firmware revision.

  • RAID cache configuration. Make sure the RAID battery is healthy and that the cache read/write balance is right for your workload.

  • Filesystem choice. XFS is a good option for streaming media. But the rates you're asking for are relatively low. Any modern Linux filesystem should be fine.

Related Topic