BTRFS single data FS still striped

btrfsraid0

UPDATE

So after a long time of research in the official documentation, I found out that btrfs allocates space in chunks each 1Gb and when using single mode, these chunks are allocated on the disk with currently the most unallocated space.
Therefore the data gets striped between the different disks and as I see it, losing one disk would result in a complete loss of files larger than 1Gb.

My conclusion is, that the only benefit of using single mode over a raid0 (also supported by btrfs) is the possibility the use different sized disks. Any other thoughts?

Original Post

So I created a btrfs filesystem consisting of 3 devices with data mode single using the following command:

sudo mkfs.btrfs -f -d single /dev/sdb7 /dev/sdb8 /dev/sdb9

Then I mounted the fs to the folder nonRaid and created a 10G file using

fallocate -l 10G nonRaid/example

I understand the difference between using raid0 and single is, that raid0 will stripe the data accross the 3 devices in my case and using single mode the data will not be striped but saved as a whole chunk on the disk as NTFS would do. Do I understand this correctly?

If so, how can one explain that when using

btrfs filesystem usage nonRaid/

it shows me the following output:

Overall:
Device size:                  59.98GiB
Device allocated:             13.02GiB
Device unallocated:           46.96GiB
Device missing:                  0.00B
Used:                         10.05GiB
Free (estimated):             47.91GiB      (min: 24.43GiB)
Data ratio:                       1.00
Metadata ratio:                   2.00
Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:11.01GiB, Used:10.05GiB
/dev/sdb7       3.01GiB  <---
/dev/sdb8       5.00GiB  <---
/dev/sdb9       3.00GiB  <---

The 10G data is split accross the 3 devices, tho not as evenly as it would be using raid0.
Does btrfs not support a mode which fills up entire disks after another?
Because this striping would still mean that if e.g. /dev/sdb8 would irrecoverably fail, my whole data of the whole filesystem would be gone – right?

Best Answer

I know this is a bit old question, though I have found it during my research on the same track. There's a pretty thorough explanation from Hugo Mills on the btrfs-kernel mailing list (https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg44523.html).

tl;dr; btrfs stores the files in 1GB chunks and allocated them equally to the available free space on each drive. So, "with equal-sized devices, the first 1 GiB will go on the first device, the second 1 GiB on the second device, and so on."

TL;DR

Btrfs' defrag feature is specific to fixing fragmentation in folder metadata and file contents, whilst the balance feature was created to "balance" (hence the name) the amount of data shared between drives whenever a drive is added or removed. While they do have some theoretical overlap in what they do, they are not directly related, thus the documentation does not link the two features.

Verbose answer below. Note of course that my long answer is in the hope that it will help others who don't have the full context of the problems faced.

Chunk Allocation

An important concept with btrfs is chunk allocation. When you write data to btrfs, it writes that data into a "current" chunk, typically 1GB in size¹. If the "current" chunk becomes full, it allocates a new chunk. If an existing chunk is emptied, its storage space is made available for re-allocation when a new chunk is needed.

If the filesystem is using more than one drive with the "dup", "single", or "raid1" storage profiles, the chunk allocator always prefers putting the next new chunk on the drive(s) with the most free space available. This ensures, generally, that drives are used equally.

How Balance Does its Thing

The balance feature works by taking existing data chunks and re-writing them into the "current" chunk. When an existing chunk is emptied in this way, it is automatically made available to the allocator. If the existing chunk being emptied was not full to start with (perhaps old data in the chunk was deleted), the net result is the freeing up of diskspace since the newer chunk is "more tightly packed" with relevant data.

This is the part which could, in theory, be used as part of a de-fragmentation strategy, which I feel is the reason that many people assume it already does. However, of course, the balance feature was built with a specific purpose in mind, thus why it does not look at the file content. It only checks whether or not the data it is taking out of the existing chunks is relevant² before copying that data to the new chunk.

Where does the Balance part come in?

When you add a new drive to the filesystem, the allocator will at first tend toward writing all new data to the new drive, mainly because it has more free space available than the existing drives. By re-writing all chunks, all initially-balanced chunks are written only to the new drive. Once it has equalised (become balanced), the rest of the data will be equally re-allocated between the drives.

Typical Balance Scenario:

I have 2x 500GB drives with 240GB used on each; I add another 500GB drive. I would typically have:

drive a: 240GB used

drive b: 240GB used

drive c: 0GB used

I start a balance of all the data. About one quarter through the balance, I'm likely to see a situation similar to the following:

drive a: 180GB used

drive b: 180GB used

drive c: 120GB used

At about the one-third mark, it appears to be balanced:

drive a: 160GB used

drive b: 160GB used

drive c: 160GB used

You can of course stop the balance operation at this point, though there are reasons (good and bad) why you might want to let it finish³.

How Fragmentation Happens in btrfs

Btrfs is a CoW (Copy on Write) filesystem, which means that data is never over-written⁴. If you have an existing 100MB file and over-write a 1MB portion of the file, that 1MB portion is not written over the existing data on the drive. Instead it is written elsewhere in the "current" chunk. Btrfs keeps track of where these "fragments" of new data are stored. This is most useful for maintaining snapshots of the data as it means the old data is preserved by default. Because SSDs, in a very similar way, also never overwrite data, this CoW mechanism lends itself well to allowing SSDs to maintain their lifespan and performance.

Where Defrag Comes In

Regardless of the advantages, some files are over-written very often (typically database files), so end up having hundreds of these fragments. With SSDs, there is little performance penalty in the short term. But with spindle drives, the performance penalty is severe.

One solution of course is to use btrfs' defrag feature. The defrag operation re-writes the file content in the current chunk in the logical order of its current state, thereby reducing the fragments into one large 100MB set of data instead of numerous separate pieces.

An alternative solution would be to use the "nocow" feature specifically for files such as this. The nocow feature causes the file to be overwritten in place. Beware that there are caveats to nocow^{5 6}.

Summary Again

The balance looks at chunks and stripes - and is not actually aware of file content except for whether or not data in those chunks is still relevant.
The defrag operation looks at folder data and individual file content and re-writes the data in as contiguous a fashion as possible. The down side is with snapshots where defrag causes duplication and extra drive usage.

Notes:

Though chunks are typically 1GB in size, they can be bigger or smaller. When using raid types, chunks are typically striped across multiple drives in 1GB multiples. For example, 5 drives with raid0 typically results in a 5GB stripe consisting of 1GB chunks being written to each drive.
Btrfs uses "references" to file content. When part of a file is overwritten, the live filesystem "references" the location where that data was written. A snapshot however might still "reference" the old location. If there is no snapshot - or the old snapshot is deleted, this results in no "ref"erences left that refer to the original overwritten content. This content is then considered irrelevant and will not be copied with the other relevant data in the balance operation.
At this point, assuming storage is using the simple "single" profile⁷, the first 160GB balanced would all be moved to the new drive - but also at this point, it still has about 320GB left to balance. The rest would be balanced equally across the drives. With spindles, ideally you would want to balance only 160 chunks before having btrfs re-balance all 3 drives for a better "spread" of the data. With SSDs, attempting to maintain an even "spread" of data gets very complicated, likely pointless, and far more likely very bad for the SSD lifespan.
The exception is the "nocow" feature.
If there are snapshots, defragmenting the "live" file causes the snapshots and "live" file to refer to divergent data locations on the disk, causing the data to be duplicated and thus take up extra diskspace. When a general-purpose de-duplication feature becomes available, this will not be as much of a problem.
Using nocow means btrfs does not maintain checksums for the file-content.
With most raid types (raid1 is the exception), "spread" across the drives is moot as the stripes are typically written across all drives anyway.

Linux – BTRFS: deleting a volume

I've run into similar issues myself using BTRFS.

First things first -- butter doesn't need to be in a partition, so unless there was some kind of unmentioned reason that you wanted it in /dev/sdb1, you did exactly what I did and ran into exactly the same problem.

After digging around and trying to find a clean solution to fixing it, wipefs is your best option -- supposedly newer versions can remove all traces. However, at the time I ran into this, I ended up just using dd to write zeros to my entire device, something like the following:

dd if=/dev/zero of=/dev/sdX bs=4M

It's the 9000 pound gorilla of solutions, but it will put your thumbdrives back to a fresh state.

SSD Warning: This might be harmful to the performance of an SSD (depending on the manufacturer) and should really only be done on thumbdrives. See this question which offers up some other alternatives (blkdiscard) that might be faster/safer/better for SSDs. This question also has some good answers that might do the equiv of this but without zeroing (secure erase feature).