Hardware RAID vs ZFS doesn't make a lot of difference from a raw throughput perspective -- either system needs to distribute data across multiple disks, and that requires running a few bit shifting operations on cached data, and scheduling writes to underlying disks. Which processor you use for that hardly matters, and synthetic workloads like running dd
can't tell you much here.
The differences are in features:
Hardware RAID is usually just a block layer, perhaps with some volume management on top, while ZFS also includes a file system layer (i.e. there is no separation of concerns in ZFS). This allows ZFS to offer compression and deduplication, while that would be hard to get right on a block layer, but for use cases where you just want a set of simple 1:1 mappings, that additional complexity will still be there.
On the other hand, hardware RAID can offer battery backed write caches that are (almost) transparent to the operating system, so it can easily compensate for the overhead of a journaling file system, and data needs to be transferred out of the CPU only once, before adding redundancy information.
Both have their use cases, and in some places, it even makes sense to combine them, e.g. with a hardware RAID controller that offers a battery backed cache, but the controller is set to JBOD mode and only re-exports the constituent disks to the operating system, which then puts ZFS on top.
In general, ZFS alone is good for "prosumer" setups, where you don't want to spend money on hardware, but still want to achieve sensible fault tolerance and some compression, and where random-access performance isn't your primary concern.
ZFS on top of JBOD is great for container and VPS hosting -- the deduplication keeps the footprint of each container small, even if they upgrade installed programs, as two containers that have installed the same upgrade get merged back into one copy of the data (which is then again kept in a redundant way).
Hardware RAID alone is good for setups where you want to add fault tolerance and a bit of caching on the outside of an existing stack -- one of the advantages of battery backed write caches is that they are maintained outside of OS control, so the controller can acknowledge a transfer as completed as soon as the data has reached the caches, and if a write is superseded later, it can be skipped, and head movements can be scheduled system-wide ignoring dependencies.
The way journaling file systems work, they will first submit a journal entry, then as soon as that is acknowledged, submit the data and after that is acknowledged, another journal entry marking the first as complete. That is a lot of head movement, especially when the disks are shared between multiple VMs that each have their own independent journaling file system, and in a busy system, the caches allow you to skip about half of the writes, but from the point of view of the inner system, the journal still behaves normally and dependent writes are performed in order.
The aspect of safely reordering dependent writes for more optimal head movements is why you want a hardware RAID at the bottom. ZFS generates dependent writes itself, so it can profit from hardware RAID too, but these are the performance bottleneck only in a limited set of use cases, mostly multi-tenant setups with little coordination between applications.
With SSDs, reordering is a lot less important, obviously, so the motivation to use hardware RAID there is mostly bulk performance -- if you've hit the point where memory and I/O interface speed on the mainboard are relevant factors, then offloading the checksum generation and transferring only a single copy one way vs multiple transfers from and to RAM (that need to be synchronized with all the other controllers in the same coherency domain) is definitely worth it. Hitting that point is a big "if" -- I haven't managed so far.
Best Answer
If you want to do this with no downtime, be SURE that you've got a good backup of the data beforehand.
You can actually accomplish this without too much hassle using the "replace member" feature/operation on a PERC 9 card. The feature is intended as a copy-back operation from a hotspare to a replacement drive, so that your configured hot spare drive "stays" as the hotspare when not in use. However, the operation can be started manually. The "replace member" operation would essentially mirror data from one of the drives in your RAID6 to a hot spare (which you should replace with a new larger capacity drive first). Since the data is mirrored, you're not throwing the entire RAID set into a degraded state each step of the way.
So the steps would look like:
Details on the "replace member" operation are on page 36 of the User's Guide
From there, you can actually expand the VD without downtime via "Online Capacity Expansion". Page 30 of the same user guide ("Reconfiguration of Virtual Disks" section) even mentions this "replace member" scenario specifically as a reason that you would have excess capacity to expand to.
Once all this is done, just perform a storage rescan from vSphere, and you should be able to expand the datastore size (though I think the VMFS filesystem might still be split into two pieces technically).
This whole process might not be the BEST way to do things, but it's certainly a Dell-supported method to get where you're going without the need for major downtime. But again, be 100% sure you've got a GOOD backup beforehand just to be on the safe side.