About minimal device number for RAID5/6

About minimal device number for RAID5/6

[Qu Wenruo]

Hi,

Recently I found that manpage of mkfs is saying minimal device number 
for RAID5 and RAID6 is 2 and 3.

Personally speaking, although I understand that RAID5/6 only requires 
1/2 devices for parity stripe, it is still quite strange behavior.

Under most case, user use raid5/6 for striping AND parity. For 2 devices 
RAID5, it's just a more expensive RAID1.

IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.

Any comment is welcomed.

Thanks,
Qu

Re: About minimal device number for RAID5/6

[Austin S. Hemmelgarn]

On 2016-08-15 03:50, Qu Wenruo wrote:
> Hi,
>
> Recently I found that manpage of mkfs is saying minimal device number
> for RAID5 and RAID6 is 2 and 3.
>
> Personally speaking, although I understand that RAID5/6 only requires
> 1/2 devices for parity stripe, it is still quite strange behavior.
>
> Under most case, user use raid5/6 for striping AND parity. For 2 devices
> RAID5, it's just a more expensive RAID1.
>
> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>
> Any comment is welcomed.
>
Based on looking at the code, we do in fact support 2/3 devices for 
raid5/6 respectively.

Personally, I agree that we should warn when trying to do this, but I 
absolutely don't think we should stop it from happening.

Re: About minimal device number for RAID5/6

[Anand Jain]

>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>>
>> Any comment is welcomed.
>>
> Based on looking at the code, we do in fact support 2/3 devices for
> raid5/6 respectively.
>
> Personally, I agree that we should warn when trying to do this, but I
> absolutely don't think we should stop it from happening.


  How does 2 disks RAID5 work ?

-Anand

Re: About minimal device number for RAID5/6

[Austin S. Hemmelgarn]

On 2016-08-15 10:08, Anand Jain wrote:
>
>
>>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>>>
>>> Any comment is welcomed.
>>>
>> Based on looking at the code, we do in fact support 2/3 devices for
>> raid5/6 respectively.
>>
>> Personally, I agree that we should warn when trying to do this, but I
>> absolutely don't think we should stop it from happening.
>
>
>  How does 2 disks RAID5 work ?
One disk is your data, the other is your parity.  In essence, it works 
like a really computationally expensive version of RAID1 with 2 disks, 
which is why it's considered a degenerate configuration.  Three disks in 
RAID6 is similar, but has a slight advantage at the moment in BTRFS 
because it's the only way to configure three disks so you can lose two 
and not lose any data as we have no support for higher order replication 
than 2 copies yet.

Re: About minimal device number for RAID5/6

[Anand Jain]

On 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
> On 2016-08-15 10:08, Anand Jain wrote:
>>
>>
>>>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>>>>
>>>> Any comment is welcomed.
>>>>
>>> Based on looking at the code, we do in fact support 2/3 devices for
>>> raid5/6 respectively.
>>>
>>> Personally, I agree that we should warn when trying to do this, but I
>>> absolutely don't think we should stop it from happening.
>>
>>
>>  How does 2 disks RAID5 work ?
> One disk is your data, the other is your parity.


>In essence, it works
> like a really computationally expensive version of RAID1 with 2 disks,
> which is why it's considered a degenerate configuration.

    How do you generate parity with only one data ?

-Anand


>  Three disks in
> RAID6 is similar, but has a slight advantage at the moment in BTRFS
> because it's the only way to configure three disks so you can lose two
> and not lose any data as we have no support for higher order replication
> than 2 copies yet.

Re: About minimal device number for RAID5/6

[Austin S. Hemmelgarn]

On 2016-08-15 10:32, Anand Jain wrote:
>
>
> On 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
>> On 2016-08-15 10:08, Anand Jain wrote:
>>>
>>>
>>>>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices
>>>>> RAID6.
>>>>>
>>>>> Any comment is welcomed.
>>>>>
>>>> Based on looking at the code, we do in fact support 2/3 devices for
>>>> raid5/6 respectively.
>>>>
>>>> Personally, I agree that we should warn when trying to do this, but I
>>>> absolutely don't think we should stop it from happening.
>>>
>>>
>>>  How does 2 disks RAID5 work ?
>> One disk is your data, the other is your parity.
>
>
>> In essence, it works
>> like a really computationally expensive version of RAID1 with 2 disks,
>> which is why it's considered a degenerate configuration.
>
>    How do you generate parity with only one data ?
You treat the data as a stripe of width 1.  That's really all there is 
to it, it's just the same as using 3 or 4 or 5 disks, just with a 
smaller stripe size.

In other systems, 4 is the minimum disk count for RAID5.  I'm not sure 
why they usually disallow 3 disks (it's perfectly legitimate usage, it's 
just almost never seen in practice (largely because nothing supports it 
and erasure coding only makes sense from an economic perspective when 
dealing with lots of data)), but they disallow 2 because it gives no 
benefit over RAID1 with 2 copies and gives worse performance, not 
because the math doesn't work with 2 disks.

Re: About minimal device number for RAID5/6

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Mon, Aug 15, 2016 at 10:32:25PM +0800, Anand Jain wrote:
> 
> 
> On 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
> >On 2016-08-15 10:08, Anand Jain wrote:
> >>
> >>
> >>>>IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
> >>>>
> >>>>Any comment is welcomed.
> >>>>
> >>>Based on looking at the code, we do in fact support 2/3 devices for
> >>>raid5/6 respectively.
> >>>
> >>>Personally, I agree that we should warn when trying to do this, but I
> >>>absolutely don't think we should stop it from happening.
> >>
> >>
> >> How does 2 disks RAID5 work ?
> >One disk is your data, the other is your parity.
> 
> 
> >In essence, it works
> >like a really computationally expensive version of RAID1 with 2 disks,
> >which is why it's considered a degenerate configuration.
> 
>    How do you generate parity with only one data ?

   For plain parity calculations, parity is the value p which solves
the expression:

x_1 XOR x_2 XOR ... XOR x_n XOR p = 0

for corresponding bits in the n data volumes. With one data volume,
n=1, and hence p = x_1.

   What's the problem? :)

   Hugo.

> -Anand
> 
> 
> > Three disks in
> >RAID6 is similar, but has a slight advantage at the moment in BTRFS
> >because it's the only way to configure three disks so you can lose two
> >and not lose any data as we have no support for higher order replication
> >than 2 copies yet.

-- 
Hugo Mills             | I always felt that as a C programmer, I was becoming
hugo@... carfax.org.uk | typecast.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: About minimal device number for RAID5/6

[Henk Slager]

On Mon, Aug 15, 2016 at 8:30 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Mon, Aug 15, 2016 at 10:32:25PM +0800, Anand Jain wrote:
>>
>>
>> On 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
>> >On 2016-08-15 10:08, Anand Jain wrote:
>> >>
>> >>
>> >>>>IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>> >>>>
>> >>>>Any comment is welcomed.
>> >>>>
>> >>>Based on looking at the code, we do in fact support 2/3 devices for
>> >>>raid5/6 respectively.
>> >>>
>> >>>Personally, I agree that we should warn when trying to do this, but I
>> >>>absolutely don't think we should stop it from happening.

About a year ago I had a raid5 array in an disk upgrade situation from
5x 2TB to 4x 4TB. As intermediate I had 2x 2TB + 2x 4TB situation for
several weeks. The 2x 2TB were getting really full and the fs was
slow. just wondering if an enospc would happen, I started an filewrite
task doing several 100 GB's and it simply did work to my surprise. At
some point, chunks only occupying the 4TB disks must have been
created. I also saw the expected write rate on the 4TB disks. CPU load
was not especially high as far as I remember, like a raid1 fs as far
as I remember.

So it is good that in such a situation, one can still use the fs. I
don't remember how the allocated/free space accounting was, probably
not correct, but I did not fill up the whole fs to see/experience
that.

I have no strong opinion whether we should warn about amount of
devices at mkfs time for raid56. It's just that the other known issues
with raid56 draw more attention.

>> >> How does 2 disks RAID5 work ?
>> >One disk is your data, the other is your parity.
>>
>>
>> >In essence, it works
>> >like a really computationally expensive version of RAID1 with 2 disks,
>> >which is why it's considered a degenerate configuration.
>>
>>    How do you generate parity with only one data ?
>
>    For plain parity calculations, parity is the value p which solves
> the expression:
>
> x_1 XOR x_2 XOR ... XOR x_n XOR p = 0
>
> for corresponding bits in the n data volumes. With one data volume,
> n=1, and hence p = x_1.
>
>    What's the problem? :)
>
>    Hugo.
>
>> -Anand
>>
>>
>> > Three disks in
>> >RAID6 is similar, but has a slight advantage at the moment in BTRFS
>> >because it's the only way to configure three disks so you can lose two
>> >and not lose any data as we have no support for higher order replication
>> >than 2 copies yet.
>
> --
> Hugo Mills             | I always felt that as a C programmer, I was becoming
> hugo@... carfax.org.uk | typecast.
> http://carfax.org.uk/  |
> PGP: E2AB1DE4          |

Re: About minimal device number for RAID5/6

[Qu Wenruo]

At 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
> On 2016-08-15 10:08, Anand Jain wrote:
>>
>>
>>>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>>>>
>>>> Any comment is welcomed.
>>>>
>>> Based on looking at the code, we do in fact support 2/3 devices for
>>> raid5/6 respectively.
>>>
>>> Personally, I agree that we should warn when trying to do this, but I
>>> absolutely don't think we should stop it from happening.
>>
>>
>>  How does 2 disks RAID5 work ?
> One disk is your data, the other is your parity.  In essence, it works
> like a really computationally expensive version of RAID1 with 2 disks,
> which is why it's considered a degenerate configuration.

I totally agree with the fact that 2 disk raid5 is just a slow raid1.

>  Three disks in
> RAID6 is similar, but has a slight advantage at the moment in BTRFS
> because it's the only way to configure three disks so you can lose two
> and not lose any data as we have no support for higher order replication
> than 2 copies yet.

It's true that btrfs doesn't support any other raid level which can 
provide 2 parities.

But the use case to gain the ability to lose 2 disks in a 3 disk raid6 
setup seems more like a trick other than normal use case.

Either in mkfs man page, or warning at mkfs time (but still allowing to 
do it), IMHO it's better to tell user "yes, you can do it, but it's not 
a really good idea"

Thanks,
Qu

Re: About minimal device number for RAID5/6

[Austin S. Hemmelgarn]

On 2016-08-15 21:32, Qu Wenruo wrote:
>
>
> At 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote:
>> On 2016-08-15 10:08, Anand Jain wrote:
>>>
>>>
>>>>> IMHO it's better to warn user about 2 devices RAID5 or 3 devices
>>>>> RAID6.
>>>>>
>>>>> Any comment is welcomed.
>>>>>
>>>> Based on looking at the code, we do in fact support 2/3 devices for
>>>> raid5/6 respectively.
>>>>
>>>> Personally, I agree that we should warn when trying to do this, but I
>>>> absolutely don't think we should stop it from happening.
>>>
>>>
>>>  How does 2 disks RAID5 work ?
>> One disk is your data, the other is your parity.  In essence, it works
>> like a really computationally expensive version of RAID1 with 2 disks,
>> which is why it's considered a degenerate configuration.
>
> I totally agree with the fact that 2 disk raid5 is just a slow raid1.
>
>>  Three disks in
>> RAID6 is similar, but has a slight advantage at the moment in BTRFS
>> because it's the only way to configure three disks so you can lose two
>> and not lose any data as we have no support for higher order replication
>> than 2 copies yet.
>
> It's true that btrfs doesn't support any other raid level which can
> provide 2 parities.
>
> But the use case to gain the ability to lose 2 disks in a 3 disk raid6
> setup seems more like a trick other than normal use case.
It absolutely is a trick, but until we have support for more than 2 
copies in raid1 mode, it is also an absolutely legitimate use case.
>
> Either in mkfs man page, or warning at mkfs time (but still allowing to
> do it), IMHO it's better to tell user "yes, you can do it, but it's not
> a really good idea"
Agreed.

Re: About minimal device number for RAID5/6

[Anand Jain]

Have a look at this..

http://www.spinics.net/lists/linux-btrfs/msg54779.html

----------
RAID5&6 devs_min values are in the context of degraded volume.
RAID1&10.. devs_min values are in the context of healthy volume.

RAID56 is correct. We already have devs_max to know the number
of devices in a healthy volumes. RAID1's devs_min is wrong so
it ended up being same as devs_max.
----------

Any comments?
Also you may use the btrfs-raid-cal simulator tool to verify.
https://github.com/asj/btrfs-raid-cal/blob/master/state-table


Thanks, Anand



On 08/15/2016 03:50 PM, Qu Wenruo wrote:
> Hi,
>
> Recently I found that manpage of mkfs is saying minimal device number
> for RAID5 and RAID6 is 2 and 3.
>
> Personally speaking, although I understand that RAID5/6 only requires
> 1/2 devices for parity stripe, it is still quite strange behavior.
>
> Under most case, user use raid5/6 for striping AND parity. For 2 devices
> RAID5, it's just a more expensive RAID1.
>
> IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6.
>
> Any comment is welcomed.
>
> Thanks,
> Qu
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html