ZFS iSCSI and free capacity in VMFS

capacityiscsitruenaszfs

I use ZFS in FreeNAS 8.0.4 and use iSCSI to provide a LUN to Vmware ESXi 5.0.

I post it at http://forums.freenas.org/showthread.php?10990-Memory-utilization-and-performance-problem/page4, but I believe people here may have more expertise on ZFS.

FreeNAS doesn't have iSCSI unmap. I have create / move / delete virtual machine between ZFS iSCSI. After a while, zpool list report 95% capacity used. However, Vmware shows VMFS only uses 50% capacity.

AFAIK, when creating a virtual machine, zvol allocated disk space.
However, when we moved / deleted virtual machine, zvol didn't update / didn't aware to update the free capacity.

[root@data1] ~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data1 2.25T 2.07T 185G 95% ONLINE /mnt

Now, I believe due to the wrong free capacity reported in zpool list, the performance is deeply affected.

How I should do to let ZFS know the real free capacity.

Thanks million.

zpool status -v

  pool: data1
 state: ONLINE
 scrub: scrub stopped after 10h31m with 0 errors on Wed Feb  6 13:32:53 2013
config:

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
        spares
          ada5p2    AVAIL   

errors: No known data errors

zpool list

NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data1  2.25T  2.08T   179G    92%  ONLINE  /mnt

Best Answer

You made your zvol too large and your ZFS implementation doesn't have the ability to unmap release blocks!

The best way to prevent this from happening is to underprovision the LUNs you present to VMware. Since we know ZFS performance degrades when the disk is near full, it's best to limit the maximum size of your zvol.

e.g. if you have 200GB zpool storage, make sure the zvols you present to VMware are less than that - maybe 150GB.

ZFS is unaware of the filesystem being using on top of the zvol block device, so all that's happening on your VMFS volume is that it knows that it has 50% free space. ZFS has no way of knowing that. You will not be able to reclaim the space without reformatting/recreating the volume or expanding the zvol.

A better solution in your case would be using NFS instead of iSCSI for this purpose. If you're not monitoring VM disk utilization and growth carefully enough to prevent this from happening with iSCSI, NFS may make more sense. I use NFS with my ZFS+VMware setups.

Also see: When using thin-provisioning with ZFS, how do you make sure you don't run out of physical disk space?

and

Incorrect Used Space reported by zVol.

Related Solutions

Missing whole disk device in OpenSolaris

Try:

devfsadm -v

otherwise, it might be an EFI partition is required. It should be created with:

format -e

fdisk -E raw-device

ZFS Thin-Provisioning – Avoid Running Out of Physical Disk Space

On the Nexenta side, there's a volume-check script that's setup to run hourly by default. It will:
Check volume health and capacity, clear correctable device errors, validate mountpoints.
It also sends a weekly summary report via email.

However, there are some things you should consider when planning a Nexenta storage solution for the purposes you've listed.

You may want to consider having multiple pools for flexibility. A single pool works, but sometimes it's necessary to move data around or just have the option of a second pool on local storage.
ZFS zvols can be expanded/contracted on the fly. For instance, if you allocate 20TB to a thin-provisioned zvol, you can change it to 30TB or 100TB very easily. You don't need to over-provision 100TB for the future if you don't have it at present.
With thin-provisioned zvols, once the space is used, you can't reclaim it. If you thin provision a 2TB zvol in a 10TB pool, fill the zvol up, then delete the VMs on that zvol, your pool will still only show 8TB free. That 2TB is going to remain.
Will you be using ZFS compression or deduplication or both? One situation where it DOES make sense to over-provision is if you're using inline compression and highly-compressible data. Same for data that is deduped. In my case, the data sets I work with compress 60%-80%, so I present larger zvols than the amount of storage I actually have.
Using mirrors versus raidz1/2/3 makes it easier to expand underlying storage. You can add mirrored disk pairs to a zpool, but you can't expand raidz1/2/3 unless you add another vdev (group of raidz(x) disks). You'd also want to rebalance the data within to redistribute across the disks.
Which virtualization technology will you be using? If VMWare, you can thin-provision. You will see datastore warnings near 80% utilization, I believe. VMware also complains if you're in a dangerous situation with snapshot size growth.
If you are doing a lot of VM testing, or have VMs that fluctuate in size, I'd suggest using iSCSI and zvols for the relatively static VMs and NFS for the test VMs (if that's an option for your preferred virtualization solution). With NFS, you can make more efficient use of your storage space since you see the zpool's full available size and don't have any size ceiling to worry about.

In short... I wouldn't overprovision to account for future growth. It's not necessary. There are hourly checks in Nexenta to alert to space utilization. Also think about whether you will use compression or not (deduplication requires a bit more planning). test things out and see what the VM footprint will look like before going into production. It will be more difficult to change afterwards.

Best Answer

Related Solutions

Missing whole disk device in OpenSolaris

ZFS Thin-Provisioning – Avoid Running Out of Physical Disk Space

Related Topic