COMSTAR and ZFS zvol giving bursty write pattern

You would need to rewrite your data to the expanded zpool in order to rebalance it. Otherwise, as time passes, your writes will be distributed across the entire pool.

VOLSIZE represents that size of the volume as it will be seen by the clients, not the size of the volume as stored on the pool.

This difference may come from multiple sources:

• space required for metadata
• space required for storing multiple copies (the "copies" parameters)
• "wasted space" due to padding while aligning blocks of "volblocksize" size to vdev structure; by vdev structure I mean two parameters: number of disks in raidz-N, and physical block size of the devices.

When creating a volume, zfs will estimate how much space it would need to use in order to be able to present a volume of "volsize" to its clients. You can see that difference in the vtest-16 and vtest-3 volumes (where refreservation is 102GB and volsize is 100GB). The calculation can be found in libzfs_dataset.c (zvol_volsize_to_reservation(uint64_t volsize, nvlist_t *props))

What is not taken into account by that calculation is the third source. The third source has little impact on vdevs that are created with disks that have 512 byte sectors. From my experiments (I have tested that by filling an entire zvol to check that out) it does make quite a difference when the vdev is created over newer 4K sector disks.

Another thing I found in my experiments is that having mirrors does not show differences between calculated refreservation and what ends up to be used.

These are my results when using 4K drives with volumes that have the default volblocksize (8K). The first column represents the number of disks in a vdev:

    raidz1  raidz2
3   135%    101%
4   148%    148%
5   162%    181%
6   162%    203%
7   171%    203%
8   171%    217%
9   181%    232%
10  181%    232%
11  181%    232%
12  181%    232%

These are my results when using 512 bytes sector drives and default volblocksize of 8K. The first column represents the number of disks in a vdev:

    raidz1  raidz2
3   101%    101%
4   104%    104%
5   101%    113%
6   105%    101%
7   108%    108%
8   110%    114%
9   101%    118%
10  102%    106%
11  103%    108%
12  104%    110%

My conclusions are the following:

• Do not use 4K drives
• If you really need to use them, create the volume using volblocksize greater than or equal to 32K; There is negligible performance impact, and negligible space usage overhead (larger block sizes require less padding to properly align).
• Prefer stripes of mirrors for your pools; This layout has both performance benefits and less space related surprises like this one.
• The estimation is clearly wrong for the cases outlined above, and this is a bug in zfs.