VMware VMFS5 and LUN sizing – multiple smaller datastores, or 1 big datastore

storagestorage-area-networkvmfsvmware-esxi

With VMFS5 no longer having the 2TB limit for a VMFS volume, I'm considering which scenario would be more beneficial overall:-

Less LUNs of larger size, or
More LUNs of smaller size.

In my case, I have a new 24-disk storage array with 600GB disks. I'll be using RAID10, so roughly 7.2TB, and trying to decide whether to go with 1 big 7TB datastore, or multiple stores of 1TB each.

What are the pros and cons of each approach?

Update: Of course, I neglected to include hot spares in my calculation, so it'll be just under 7.2TB, but the general idea is the same. 🙂

Update 2: There are 60 VMs and 3 hosts. None of our VMs are particularly I/O intensive. Most of them are web/app servers, and also things like monitoring (munin/nagios), 2 Windows DCs with minimal load, and so on. DB servers are rarely virtualised unless they have VERY low I/O requirements. Right now I think the only virtual DB server we have is an MSSQL box and the DB on that box is <1GB.

Update 3: Some more info on the array and FC connectivity. The array is an IBM DS3524, 2 controllers with 2GB cache each. 4x 8Gbit FC ports per controller. Each ESXi host has 2x 4Gbit FC HBAs.

Best Answer

You didn't specify how many VMs you have or what they're going to be doing. Even without that information, I'd avoid one making one big LUN for blocksize/performance, contention and flexibility reasons.

Related Solutions

Assign and remove multiple LUNs from VMs on ESXi 4

I can't see why you shouldn't be able to do this with any standard ESX\ESXi setup.

You should be able to do it with Raw Device Mappings. Assuming the zoning and LUN presentation procedures on the SAN end remains unchanged you can use vmkfstools to tell ESX to rescan and detect the new LUNs at the host level. Once that is found you can then either create VMDK's and add them to the VM or present the entire volumes as Raw Device Mappings (RDM's).

With ESX you could script this on the Service Console command line but for ESXi you will need to use either the PowerCLI (Powershell) or the Perl CLI tools. The vSphere Management Appliance is a Linux appliance that has all of the tools pre-packed if you want to take that route. The documentation for all three CLI approaches can be found here.

The general outline of what you will want to do is:

1. Rescan for the new LUNs on the host.

vicfg-rescan [vmhbaX]

You will probably just want to scan the relevant HBA's that the LUN is presented to so substitute vmhbaX with the relevant HBA names that are connected to your SAN.

2. Create an RDM stub that maps to the new LUN

vmkfstools –a lsilogic -r /vmfs/devices/disks/vmhba1:0:0:0 /vmfs/volumes/storage1/testluns/testrdmlun.vmdk

You will need to figure out the LUN reference for your LUNS and set the vmdk to a location and name that makes sense in your environment. There are a couple of syntax variants with this and I haven't used this on ESXi 4 but this format used to work fine for me on 3.5. There are two RDM modes, if you need more SCSI functionality then Raw Device Mapping Passthrough mode may be more appropriate for you, in that case replace the -r with a -x.

3. Present the new disk to your VM(s).

Once you have the disk prepared in this way there are a couple of ways to present these to the OS within the VM. You can edit the VM config and add an entry for this device, or you could have this specific target vmdk already configured in a VM and you could run through the above discovery steps while the VM was powered off. If you want a more dynamic mechanism the best way to do what you want to do would be to use the VMware Disk Mount utility - this allows you to directly mount the RDM (or any other VMDK) from within the guest OS without having to mess with the VM config.

If you are using the remote CLI for the rescan and vmkfstools parts you may have to specify the target host and authentication credentials as part of the commands.

The same approach could be used with standard VMDK files but you would need to format the LUN as a VMFS first and then create a suitable VMDK on it. As far as I can tell from your description there is no benefit to you to be gained from doing it that way.

ESX hosts lose connectivity with iSCSI SAN LUNs

Let's try a little bit more complicated way. Try to use some other iSCSI solution to check if it's a ESX trouble or, iSCSI itself.

I'll redcomend you StarWind. You can download trial there.

Related Topic