You may want to consider using another disk measurement tool. Have The p400 controller and HP SAS disks are very common setups, and if you were only receiving 5MB/s random reads/writes, there may be a larger problem. Can you run your analysis using another disk performance analysis program like Iometer?
For the SSD drives, I'd expect a better random read/write, but it helps to understand exactly what you're testing.
This is several questions in one, so I'll try to address a few of them.
I typically set Smart Array controllers to leverage a higher write cache ratio. I prefer to have 75% write cache because the OS (using the XFS filesystem) caches aggressively. XFS will make a difference, but what are you tuning for? Are you tuning to simply achieve specific numbers, or is there an application requirement driving this?
ext3 isn't the fastest filesystem out there. But you have some mount options (e.g. noatime) and journal settings you could tweak.
I don't use LVM, especially with HP controllers that can provide many of the same benefits.
You have I/O scheduler and elevator settings (e.g. noop or deadline, in this case) that can be tuned, but that's a function of your application's actual needs.
If you do go with XFS, try a basic config then try some advanced configuration settings. Over time, I've ended up with mount parameters very similar to the one in the original link.
I just ran the following iozone command line on an XFS partition contained within a DL380 G5 with P400i, 12GB RAM and 8 x 146GB 10k drives. The elevator is set to deadline:
Command line used: iozone -t1 -i0 -i1 -i2 -r1m -s24g
initial writers = 348957.75 KB/sec
rewriters = 335130.03 KB/sec
readers = 132851.70 KB/sec
re-readers = 137116.27 KB/sec
random readers = 35774.41 KB/sec
random writers = 250618.38 KB/sec
Best Answer
Yep.
Try to keep the controllers on similar firmware levels and use the same drive positions on the new server. But HP Smart Array RAID metadata lives on the disks, not the controllers.
Also this:
Move Raidset from HP Smart Array P400 to E200
moving raid 10 to another identical server both on Smart Array 6i controllers