How to use a file as a zpool

First of all, for general ZFS best practices the Solaris Internals wiki is quite useful: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

If you want, you can create a separate ZFS pool composed entirely of SSDs. However, this will not improve the performance of any other pools. A more cost-effective approach for most workloads is to use the hybrid storage pool feature; that is, adding SSDs to normal spinning disk in order to improve read and/or write performance.

Key points:

• SSDs can be added in to an existing pool to improve the performance of that pool only. They don't have different names - they are just devices within the pool. You can add multiple cache and log devices to the pool depending on your performance requirements.

• To improve the read performance of the pool, assign SSDs as cache devices ("L2ARC").

• To improve the write performance of the pool, assign SSDs as log devices ("ZIL").

The L2ARC is a read cache that fills up over time and stores data based on a combination of which blocks are most frequently used and most recently used. This provides fast access to "hot" data. It gives almost the speed benefit of RAM at a lower cost per gigabyte.

The ZIL is an intent-log - adding SSDs allows synchronous writes to happen at high IOPS and very low latency, which improves application performance in some cases (e.g. databases and NFS shares).

Unfortunately ZFS does not currently support shrinking the pool.

Since you already mentioned that your intention is to destroy and recreate the pool, here's what to do. But first...

WARNING: DESTROYING YOUR POOLS WILL DESTROY YOUR DATA. YOUR BLOOD IS ON YOUR OWN HEAD.

Now, with that out of the way:

1. Create a new pool mounted on /usr-new
2. Use zfs-send from a snapshot of usr-pool to usr-new (You also need to account for any descendent datasets).
3. zfs unmount usr-new.
4. Change the mountpoint of usr-new to /usr (it may complain, but should work. Check with zfs get mountpoint usr-new)
5. Set usr-pool (i.e., the old one) to use legacy mounting with zfs set mountpoint=legacy usr-pool (this will cause it to not be auto mounted at boot but also not immediately unmount it).
6. Reboot.

You'll boot up with the new usr-new mounted on /usr and the old usr-pool not mounted. You are then free to destroy usr-pool and reuse disks as necessary.