BTRFS and Luks integration

From the sounds of it, you're running into this:

Allocation is done on a round-robin basis. If you have a raid1 strategy on a volume made up of mismatched drives (volumes of differing sizes), your smaller volume may fill up while leaving lots of space free on your single largest drive. You can verify that this is an issue if there is any discrepancy between 'df' and 'btrfs filesystem df [mountpoint]' AND if the latter command also shows that "total" and "used" are the same on the "Data" line. A rebalance may mitigate this problem. (2.6.33)

• If your volume does fill up in this manner, a rebalance may quickly cause an ENOSPC ("Error NO SPaCe left on device") oops. You may have to delete a relatively large file to resolve this impasse, then a rebalance will succeed. (2.6.33)

https://btrfs.wiki.kernel.org/index.php/Gotchas

Emphassis mine. The second partition on a third 80GB drive you mention is probably filling up well before the pair of 250GB drives are, and it's triggering this particular Gotcha.

Also, BTrFS is a beta filesystem for a reason.

The bad recorded performances stem from different factors:

• mechanical disks are simply very bad at random read/write IO. To discover how bad they can be, simply append --sync=1 to your fio command (short story: they are incredibly bad, at least when compared to proper BBU RAID controllers or powerloss-protected SSDs);

• RAID5 has an inherent write penalty due to stripe read/modify/write. Moreover it is strongly suggested to avoid it on multi-TB mechanical disks due to safety reasons. Having 4 disks, please seriously consider using RAID10 instead;

• LUKS, providing software-based full-disk encryption, inevitably has its (significant) toll on both reads and writes;

• using BTRFS, LVM is totally unnecessary. While a fat LVM-based volume will not impair performance in any meaningful way by itself, you are nonetheless inserting another IO layer and exposing yourself to (more) alignment issues;

• finally, BTRFS itself is no particularly fast. Especially your slow sequential reads can be tracked to BTRFS horrible fragmentation (due it being CoW and enforcing 4K granularity - as a comparison, to obtain good performance from ZFS one should generally select 64K-128K records when using mechanical disks).

To have a baseline performance comparison, I strongly suggest redoing your IO stack measuring random & sequential read/write speed at each step. In other words:

• create a RAID10 array and run dd and fio on the raw array (without a filesystem);

• if full-disk encryption is really needed, use LUKS to create an encrypted device and re-run dd + fio on the raw encrypted device (again, with no filesystem). Compare to previous results to have an idea of what it means performance-wise;

• try both XFS and BTRFS (running the usual dd + fio quick bench) to understand how two different filesystems behave. If BTRFS is too slow, try replacing it with lvmthin and XFS (but remember that in this case you will lose user data checksum, for which you need yet another layer - dmintegrity - itself commanding a significant performance hit).

If all this seems a mess, well, it really is so. By doing all the above you are just scratching storage performance: one had to consider real application behavior (rather than totally sequential dd or pure random fio results), cache hit ratio, IO pattern alignment, etc. But hey - few is much better than nothing, so lets start with something basic.