Re: Safety of raid1 vs raid10

[Qu Wenruo <wqu@suse.com>]

在 2024/12/10 17:06, Scoopta 写道:
> Yeah, that makes a lot of sense. Given what I know about btrfs it seemed 
> very unlikely it would pair up drives consistently. This actually gives 
> me an interesting side question. With raid1, which is not striped, is 
> there any guarantee that a file will be placed contiguously on a given 
> drive or is the only guarantee for chunks while files over 1GiB, and 
> therefore occupying more than one chunk, could be spread across multiple 
> drives so long as each chunk is mirrored?

For a file, there is no guarantee at all that it will be placed on a 
given drive.

As you mentioned, a file can exist on multiple chunks, thus it can be 
spread across different devices.

And the guarantee is, as long as all the file extents are all on raid1 
chunks, all of these file extents will have two copies on different devices.



And to be more accurate, a file is consisted of zero or more file extents.
The file extent size can vary, from the block size (usually 4K for 
btrfs, and we normally call it sector size), to as large as 128M 
(non-compressed extent) or 128K (compressed one).

And the extent size is determined by various factors, from the free 
space of the fs, to the write pattern (worst case like checker board 
writes, 4K write then 4K hole, will definitely result 4K sized extents).

So even for a file smaller than chunk size, it can have multiple file 
extents on different chunks.

Thus there isn't really any guarantee on how a file is stored where, due 
to all the layers involved:

   file -> file extents -> chunks


And I forgot a corner case, inlined file extents, which is fully stored 
inside a tree block, can have a different profile than the data profile 
completely.

That's why we have fiemap ioctl, to show the file extents layout (inside 
btrfs logical address space), then only with the chunk layout (btrfs ins 
dump-tree or btrfs-map-logical) info, one can really determine where the 
data is.

And if you want to dig this deep, welcome to the rabbit whole of how to 
read a file on btrfs, and there is also a small project explaining the 
whole process: https://github.com/adam900710/btrfs-fuse

Thanks,
Qu

> 
> On 12/9/24 8:36 PM, Qu Wenruo wrote:
>>
>>
>> 在 2024/12/10 12:56, Scoopta 写道:
>>> I've read online that btrfs raid10 is theoretically safer than raid1 
>>> because raid10 groups drives together into mirrored pairs making the 
>>> filesystem more likely to successfully survive a multi-drive failure 
>>> event.
>>
>> It's only theoretically possible, but hardly possible in the real world.
>>
>>
>> For one single RAID10 chunk, btrfs can tolerant as many as half of the 
>> devices being missing, as long as each sub stripe (the RAID1 pair) has 
>> one device standing.
>>
>> E.g. for chunk at bytenr X, we have 4 stripes:
>>
>>  stripe 0 devid 1 physical X1
>>  stripe 1 devid 2 physical X2
>>  stripe 2 devid 3 physical X3
>>  stripe 3 devid 4 physical X4
>>
>> We can have either devid 1+3 or devid 2+4 missing, and btrfs is 
>> totally fine with that chunk.
>>
>> But the real problem is, one btrfs has more than 3 chunks, and 
>> normally one chunk is only 1GiB in size, so for a btrfs with 1TiB used 
>> space, it will have at least 1024 chunks.
>>
>> Good luck all the chunks have the same stripe layout.
>>
>> If there is another chunk at bytenr Y, also 4 stripes but a different 
>> layout:
>>
>>  stripe 0 devid 1 physical Y1
>>  stripe 1 devid 3 physical Y2
>>  stripe 2 devid 2 physical Y3
>>  stripe 3 devid 4 physical Y4
>>
>> Then the devid 1+3 missing is fine for chunk X, but not for chunk Y.
>>
>> In really, the chunk layout is never ensured, and I just did the same 
>> RAID10 assumption in my btrfs-fuse project, until it failed selftest 
>> (missing two devices for a RAID10 btrfs) on a recent kernel, exactly 
>> due to the device rotation.
>>
>> Thanks,
>> Qu
>>
>>> I can't find any documentation that says this to be the case. Is it 
>>> true that btrfs pairs drives together for raid10 but not raid1, if 
>>> this is the case what's the reasoning for it?