Safety of raid1 vs raid10

Safety of raid1 vs raid10

[Scoopta]

I've read online that btrfs raid10 is theoretically safer than raid1 
because raid10 groups drives together into mirrored pairs making the 
filesystem more likely to successfully survive a multi-drive failure 
event. I can't find any documentation that says this to be the case. Is 
it true that btrfs pairs drives together for raid10 but not raid1, if 
this is the case what's the reasoning for it?

Re: Safety of raid1 vs raid10

[Zygo Blaxell]

On Mon, Dec 09, 2024 at 06:26:24PM -0800, Scoopta wrote:
> I've read online that btrfs raid10 is theoretically safer than raid1 because
> raid10 groups drives together into mirrored pairs making the filesystem more
> likely to successfully survive a multi-drive failure event. I can't find any
> documentation that says this to be the case. Is it true that btrfs pairs
> drives together for raid10 but not raid1, if this is the case what's the
> reasoning for it?

It is _possible_ for raid10 and raid1 to be arranged such that multiple drive
losses are possible.  e.g. if all odd numbered devices are paired with all
even numbered devices, then any odd numbered device can be lost and the
filesystem still survives.

This is not _guaranteed_, it is only _possible_.

In a filesystem with an odd number of drives, or with drives of varying
sizes, the block groups will only guarantee that one drive loss can be
tolerated in each block group, in order to have the flexibility needed
to fill all available space.  In such cases it is common that block
groups are arranged in such a way that loss of any two drives will break
the filesystem.

Re: Safety of raid1 vs raid10

[Scoopta]

Ok, this is what I thought but I have seen so many articles that imply 
raid10 groups drives and I have found no evidence for that. I was always 
under the impression that neither raid 1 nor 10 grouped drives together 
intentionally and any grouping was circumstantial.

On 12/9/24 6:42 PM, Zygo Blaxell wrote:
> On Mon, Dec 09, 2024 at 06:26:24PM -0800, Scoopta wrote:
>> I've read online that btrfs raid10 is theoretically safer than raid1 because
>> raid10 groups drives together into mirrored pairs making the filesystem more
>> likely to successfully survive a multi-drive failure event. I can't find any
>> documentation that says this to be the case. Is it true that btrfs pairs
>> drives together for raid10 but not raid1, if this is the case what's the
>> reasoning for it?
> It is _possible_ for raid10 and raid1 to be arranged such that multiple drive
> losses are possible.  e.g. if all odd numbered devices are paired with all
> even numbered devices, then any odd numbered device can be lost and the
> filesystem still survives.
>
> This is not _guaranteed_, it is only _possible_.
>
> In a filesystem with an odd number of drives, or with drives of varying
> sizes, the block groups will only guarantee that one drive loss can be
> tolerated in each block group, in order to have the flexibility needed
> to fill all available space.  In such cases it is common that block
> groups are arranged in such a way that loss of any two drives will break
> the filesystem.

Re: Safety of raid1 vs raid10

[Zygo Blaxell]

On Mon, Dec 09, 2024 at 06:44:14PM -0800, Scoopta wrote:
> Ok, this is what I thought but I have seen so many articles that imply
> raid10 groups drives and I have found no evidence for that. I was always
> under the impression that neither raid 1 nor 10 grouped drives together
> intentionally and any grouping was circumstantial.

In traditional, non-btrfs raid1/raid10 setups, this can easily be the
case, and people might be assuming this is also true on btrfs without
checking.

Traditional setups are much less flexible on space--you can't have
different sizes of drives--but you can organize the drives such that
each of them is entirely mirroring one other drive.  This can tolerate
up to N/2 failures, as long as each failure only removes one drive from
each pair--but it can also fail completely with only 2 drive failures,
if they're both mirrors of the same stripe.

> On 12/9/24 6:42 PM, Zygo Blaxell wrote:
> > On Mon, Dec 09, 2024 at 06:26:24PM -0800, Scoopta wrote:
> > > I've read online that btrfs raid10 is theoretically safer than raid1 because
> > > raid10 groups drives together into mirrored pairs making the filesystem more
> > > likely to successfully survive a multi-drive failure event. I can't find any
> > > documentation that says this to be the case. Is it true that btrfs pairs
> > > drives together for raid10 but not raid1, if this is the case what's the
> > > reasoning for it?
> > It is _possible_ for raid10 and raid1 to be arranged such that multiple drive
> > losses are possible.  e.g. if all odd numbered devices are paired with all
> > even numbered devices, then any odd numbered device can be lost and the
> > filesystem still survives.
> > 
> > This is not _guaranteed_, it is only _possible_.
> > 
> > In a filesystem with an odd number of drives, or with drives of varying
> > sizes, the block groups will only guarantee that one drive loss can be
> > tolerated in each block group, in order to have the flexibility needed
> > to fill all available space.  In such cases it is common that block
> > groups are arranged in such a way that loss of any two drives will break
> > the filesystem.
> 

Re: Safety of raid1 vs raid10

[Scoopta]

On 12/9/24 9:12 PM, Zygo Blaxell wrote:

> On Mon, Dec 09, 2024 at 06:44:14PM -0800, Scoopta wrote:
>> Ok, this is what I thought but I have seen so many articles that imply
>> raid10 groups drives and I have found no evidence for that. I was always
>> under the impression that neither raid 1 nor 10 grouped drives together
>> intentionally and any grouping was circumstantial.
> In traditional, non-btrfs raid1/raid10 setups, this can easily be the
> case, and people might be assuming this is also true on btrfs without
> checking.

Yeah I figured that's where the misconception was coming from but I 
wanted to verify that it was a misconception. It didn't make sense to me 
that btrfs would pair up raid10 drives but not raid1 drives. I figured 
they'd all behave the same way and that btrfs was free to organize data 
however it wanted provided the redundancy "on the tin", so to speak, was 
met.


> Traditional setups are much less flexible on space--you can't have
> different sizes of drives--but you can organize the drives such that
> each of them is entirely mirroring one other drive.  This can tolerate
> up to N/2 failures, as long as each failure only removes one drive from
> each pair--but it can also fail completely with only 2 drive failures,
> if they're both mirrors of the same stripe.
>
>> On 12/9/24 6:42 PM, Zygo Blaxell wrote:
>>> On Mon, Dec 09, 2024 at 06:26:24PM -0800, Scoopta wrote:
>>>> I've read online that btrfs raid10 is theoretically safer than raid1 because
>>>> raid10 groups drives together into mirrored pairs making the filesystem more
>>>> likely to successfully survive a multi-drive failure event. I can't find any
>>>> documentation that says this to be the case. Is it true that btrfs pairs
>>>> drives together for raid10 but not raid1, if this is the case what's the
>>>> reasoning for it?
>>> It is _possible_ for raid10 and raid1 to be arranged such that multiple drive
>>> losses are possible.  e.g. if all odd numbered devices are paired with all
>>> even numbered devices, then any odd numbered device can be lost and the
>>> filesystem still survives.
>>>
>>> This is not _guaranteed_, it is only _possible_.
>>>
>>> In a filesystem with an odd number of drives, or with drives of varying
>>> sizes, the block groups will only guarantee that one drive loss can be
>>> tolerated in each block group, in order to have the flexibility needed
>>> to fill all available space.  In such cases it is common that block
>>> groups are arranged in such a way that loss of any two drives will break
>>> the filesystem.

Re: Safety of raid1 vs raid10

[Qu Wenruo]

在 2024/12/10 12:56, Scoopta 写道:
> I've read online that btrfs raid10 is theoretically safer than raid1 
> because raid10 groups drives together into mirrored pairs making the 
> filesystem more likely to successfully survive a multi-drive failure 
> event.

It's only theoretically possible, but hardly possible in the real world.


For one single RAID10 chunk, btrfs can tolerant as many as half of the 
devices being missing, as long as each sub stripe (the RAID1 pair) has 
one device standing.

E.g. for chunk at bytenr X, we have 4 stripes:

  stripe 0 devid 1 physical X1
  stripe 1 devid 2 physical X2
  stripe 2 devid 3 physical X3
  stripe 3 devid 4 physical X4

We can have either devid 1+3 or devid 2+4 missing, and btrfs is totally 
fine with that chunk.

But the real problem is, one btrfs has more than 3 chunks, and normally 
one chunk is only 1GiB in size, so for a btrfs with 1TiB used space, it 
will have at least 1024 chunks.

Good luck all the chunks have the same stripe layout.

If there is another chunk at bytenr Y, also 4 stripes but a different 
layout:

  stripe 0 devid 1 physical Y1
  stripe 1 devid 3 physical Y2
  stripe 2 devid 2 physical Y3
  stripe 3 devid 4 physical Y4

Then the devid 1+3 missing is fine for chunk X, but not for chunk Y.

In really, the chunk layout is never ensured, and I just did the same 
RAID10 assumption in my btrfs-fuse project, until it failed selftest 
(missing two devices for a RAID10 btrfs) on a recent kernel, exactly due 
to the device rotation.

Thanks,
Qu

> I can't find any documentation that says this to be the case. Is 
> it true that btrfs pairs drives together for raid10 but not raid1, if 
> this is the case what's the reasoning for it?

Re: Safety of raid1 vs raid10

[Scoopta]

Yeah, that makes a lot of sense. Given what I know about btrfs it seemed 
very unlikely it would pair up drives consistently. This actually gives 
me an interesting side question. With raid1, which is not striped, is 
there any guarantee that a file will be placed contiguously on a given 
drive or is the only guarantee for chunks while files over 1GiB, and 
therefore occupying more than one chunk, could be spread across multiple 
drives so long as each chunk is mirrored?

On 12/9/24 8:36 PM, Qu Wenruo wrote:
>
>
> 在 2024/12/10 12:56, Scoopta 写道:
>> I've read online that btrfs raid10 is theoretically safer than raid1 
>> because raid10 groups drives together into mirrored pairs making the 
>> filesystem more likely to successfully survive a multi-drive failure 
>> event.
>
> It's only theoretically possible, but hardly possible in the real world.
>
>
> For one single RAID10 chunk, btrfs can tolerant as many as half of the 
> devices being missing, as long as each sub stripe (the RAID1 pair) has 
> one device standing.
>
> E.g. for chunk at bytenr X, we have 4 stripes:
>
>  stripe 0 devid 1 physical X1
>  stripe 1 devid 2 physical X2
>  stripe 2 devid 3 physical X3
>  stripe 3 devid 4 physical X4
>
> We can have either devid 1+3 or devid 2+4 missing, and btrfs is 
> totally fine with that chunk.
>
> But the real problem is, one btrfs has more than 3 chunks, and 
> normally one chunk is only 1GiB in size, so for a btrfs with 1TiB used 
> space, it will have at least 1024 chunks.
>
> Good luck all the chunks have the same stripe layout.
>
> If there is another chunk at bytenr Y, also 4 stripes but a different 
> layout:
>
>  stripe 0 devid 1 physical Y1
>  stripe 1 devid 3 physical Y2
>  stripe 2 devid 2 physical Y3
>  stripe 3 devid 4 physical Y4
>
> Then the devid 1+3 missing is fine for chunk X, but not for chunk Y.
>
> In really, the chunk layout is never ensured, and I just did the same 
> RAID10 assumption in my btrfs-fuse project, until it failed selftest 
> (missing two devices for a RAID10 btrfs) on a recent kernel, exactly 
> due to the device rotation.
>
> Thanks,
> Qu
>
>> I can't find any documentation that says this to be the case. Is it 
>> true that btrfs pairs drives together for raid10 but not raid1, if 
>> this is the case what's the reasoning for it?

Re: Safety of raid1 vs raid10

[Qu Wenruo]

在 2024/12/10 17:06, Scoopta 写道:
> Yeah, that makes a lot of sense. Given what I know about btrfs it seemed 
> very unlikely it would pair up drives consistently. This actually gives 
> me an interesting side question. With raid1, which is not striped, is 
> there any guarantee that a file will be placed contiguously on a given 
> drive or is the only guarantee for chunks while files over 1GiB, and 
> therefore occupying more than one chunk, could be spread across multiple 
> drives so long as each chunk is mirrored?

For a file, there is no guarantee at all that it will be placed on a 
given drive.

As you mentioned, a file can exist on multiple chunks, thus it can be 
spread across different devices.

And the guarantee is, as long as all the file extents are all on raid1 
chunks, all of these file extents will have two copies on different devices.



And to be more accurate, a file is consisted of zero or more file extents.
The file extent size can vary, from the block size (usually 4K for 
btrfs, and we normally call it sector size), to as large as 128M 
(non-compressed extent) or 128K (compressed one).

And the extent size is determined by various factors, from the free 
space of the fs, to the write pattern (worst case like checker board 
writes, 4K write then 4K hole, will definitely result 4K sized extents).

So even for a file smaller than chunk size, it can have multiple file 
extents on different chunks.

Thus there isn't really any guarantee on how a file is stored where, due 
to all the layers involved:

   file -> file extents -> chunks


And I forgot a corner case, inlined file extents, which is fully stored 
inside a tree block, can have a different profile than the data profile 
completely.

That's why we have fiemap ioctl, to show the file extents layout (inside 
btrfs logical address space), then only with the chunk layout (btrfs ins 
dump-tree or btrfs-map-logical) info, one can really determine where the 
data is.

And if you want to dig this deep, welcome to the rabbit whole of how to 
read a file on btrfs, and there is also a small project explaining the 
whole process: https://github.com/adam900710/btrfs-fuse

Thanks,
Qu

> 
> On 12/9/24 8:36 PM, Qu Wenruo wrote:
>>
>>
>> 在 2024/12/10 12:56, Scoopta 写道:
>>> I've read online that btrfs raid10 is theoretically safer than raid1 
>>> because raid10 groups drives together into mirrored pairs making the 
>>> filesystem more likely to successfully survive a multi-drive failure 
>>> event.
>>
>> It's only theoretically possible, but hardly possible in the real world.
>>
>>
>> For one single RAID10 chunk, btrfs can tolerant as many as half of the 
>> devices being missing, as long as each sub stripe (the RAID1 pair) has 
>> one device standing.
>>
>> E.g. for chunk at bytenr X, we have 4 stripes:
>>
>>  stripe 0 devid 1 physical X1
>>  stripe 1 devid 2 physical X2
>>  stripe 2 devid 3 physical X3
>>  stripe 3 devid 4 physical X4
>>
>> We can have either devid 1+3 or devid 2+4 missing, and btrfs is 
>> totally fine with that chunk.
>>
>> But the real problem is, one btrfs has more than 3 chunks, and 
>> normally one chunk is only 1GiB in size, so for a btrfs with 1TiB used 
>> space, it will have at least 1024 chunks.
>>
>> Good luck all the chunks have the same stripe layout.
>>
>> If there is another chunk at bytenr Y, also 4 stripes but a different 
>> layout:
>>
>>  stripe 0 devid 1 physical Y1
>>  stripe 1 devid 3 physical Y2
>>  stripe 2 devid 2 physical Y3
>>  stripe 3 devid 4 physical Y4
>>
>> Then the devid 1+3 missing is fine for chunk X, but not for chunk Y.
>>
>> In really, the chunk layout is never ensured, and I just did the same 
>> RAID10 assumption in my btrfs-fuse project, until it failed selftest 
>> (missing two devices for a RAID10 btrfs) on a recent kernel, exactly 
>> due to the device rotation.
>>
>> Thanks,
>> Qu
>>
>>> I can't find any documentation that says this to be the case. Is it 
>>> true that btrfs pairs drives together for raid10 but not raid1, if 
>>> this is the case what's the reasoning for it?

Re: Safety of raid1 vs raid10

[Scoopta]

Hmmmm, very interesting. I do love this rabbit hole I've ended up down 
although the data structures involved with modern filesystems hurt my 
head a bit. My experience implementing a filesystem starts and ends with 
ext2 as it's pretty simple data structure wise. I'll give some of the 
code a read and see if I can get my head around it.

On 12/9/24 11:54 PM, Qu Wenruo wrote:
>
>
> 在 2024/12/10 17:06, Scoopta 写道:
>> Yeah, that makes a lot of sense. Given what I know about btrfs it 
>> seemed very unlikely it would pair up drives consistently. This 
>> actually gives me an interesting side question. With raid1, which is 
>> not striped, is there any guarantee that a file will be placed 
>> contiguously on a given drive or is the only guarantee for chunks 
>> while files over 1GiB, and therefore occupying more than one chunk, 
>> could be spread across multiple drives so long as each chunk is 
>> mirrored?
>
> For a file, there is no guarantee at all that it will be placed on a 
> given drive.
>
> As you mentioned, a file can exist on multiple chunks, thus it can be 
> spread across different devices.
>
> And the guarantee is, as long as all the file extents are all on raid1 
> chunks, all of these file extents will have two copies on different 
> devices.
>
>
>
> And to be more accurate, a file is consisted of zero or more file 
> extents.
> The file extent size can vary, from the block size (usually 4K for 
> btrfs, and we normally call it sector size), to as large as 128M 
> (non-compressed extent) or 128K (compressed one).
>
> And the extent size is determined by various factors, from the free 
> space of the fs, to the write pattern (worst case like checker board 
> writes, 4K write then 4K hole, will definitely result 4K sized extents).
>
> So even for a file smaller than chunk size, it can have multiple file 
> extents on different chunks.
>
> Thus there isn't really any guarantee on how a file is stored where, 
> due to all the layers involved:
>
>   file -> file extents -> chunks
>
>
> And I forgot a corner case, inlined file extents, which is fully 
> stored inside a tree block, can have a different profile than the data 
> profile completely.
>
> That's why we have fiemap ioctl, to show the file extents layout 
> (inside btrfs logical address space), then only with the chunk layout 
> (btrfs ins dump-tree or btrfs-map-logical) info, one can really 
> determine where the data is.
>
> And if you want to dig this deep, welcome to the rabbit whole of how 
> to read a file on btrfs, and there is also a small project explaining 
> the whole process: https://github.com/adam900710/btrfs-fuse
>
> Thanks,
> Qu
>
>>
>> On 12/9/24 8:36 PM, Qu Wenruo wrote:
>>>
>>>
>>> 在 2024/12/10 12:56, Scoopta 写道:
>>>> I've read online that btrfs raid10 is theoretically safer than 
>>>> raid1 because raid10 groups drives together into mirrored pairs 
>>>> making the filesystem more likely to successfully survive a 
>>>> multi-drive failure event.
>>>
>>> It's only theoretically possible, but hardly possible in the real 
>>> world.
>>>
>>>
>>> For one single RAID10 chunk, btrfs can tolerant as many as half of 
>>> the devices being missing, as long as each sub stripe (the RAID1 
>>> pair) has one device standing.
>>>
>>> E.g. for chunk at bytenr X, we have 4 stripes:
>>>
>>>  stripe 0 devid 1 physical X1
>>>  stripe 1 devid 2 physical X2
>>>  stripe 2 devid 3 physical X3
>>>  stripe 3 devid 4 physical X4
>>>
>>> We can have either devid 1+3 or devid 2+4 missing, and btrfs is 
>>> totally fine with that chunk.
>>>
>>> But the real problem is, one btrfs has more than 3 chunks, and 
>>> normally one chunk is only 1GiB in size, so for a btrfs with 1TiB 
>>> used space, it will have at least 1024 chunks.
>>>
>>> Good luck all the chunks have the same stripe layout.
>>>
>>> If there is another chunk at bytenr Y, also 4 stripes but a 
>>> different layout:
>>>
>>>  stripe 0 devid 1 physical Y1
>>>  stripe 1 devid 3 physical Y2
>>>  stripe 2 devid 2 physical Y3
>>>  stripe 3 devid 4 physical Y4
>>>
>>> Then the devid 1+3 missing is fine for chunk X, but not for chunk Y.
>>>
>>> In really, the chunk layout is never ensured, and I just did the 
>>> same RAID10 assumption in my btrfs-fuse project, until it failed 
>>> selftest (missing two devices for a RAID10 btrfs) on a recent 
>>> kernel, exactly due to the device rotation.
>>>
>>> Thanks,
>>> Qu
>>>
>>>> I can't find any documentation that says this to be the case. Is it 
>>>> true that btrfs pairs drives together for raid10 but not raid1, if 
>>>> this is the case what's the reasoning for it?
>