Creating a 3-disk RAID6 array

Creating a 3-disk RAID6 array

[Oliver Martin]

Hi,

is there any specific reason why md refuses to create a RAID6 array with 
3 disks? My (probably naive) understanding suggests it should be the 
same as a 3-disk RAID1, similar to a 2-disk RAID5.

The reason I'm asking is that I currently have space on three disks for 
a new array, and would like to expand it when I add a fourth. I tried 
this scenario with a few loopback devices, but the only way to go from a 
3-disk RAID1 to a 4-disk RAID6 seems to be via an intermediate 3-disk 
RAID5, requiring two reshapes. I'd like to avoid one of them, if at all 
possible.

Regards,
Oliver

Re: Creating a 3-disk RAID6 array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Thu, 17 May 2012 03:24:26 +0200 Oliver Martin <oliver@volatilevoid.net>
wrote:

> Hi,
> 
> is there any specific reason why md refuses to create a RAID6 array with 
> 3 disks? My (probably naive) understanding suggests it should be the 
> same as a 3-disk RAID1, similar to a 2-disk RAID5.

I'll have to leave for for hpa to answer.  I've occasionally thought that
maybe it should be fixed, but it never seemed worth the effort.

> 
> The reason I'm asking is that I currently have space on three disks for 
> a new array, and would like to expand it when I add a fourth. I tried 
> this scenario with a few loopback devices, but the only way to go from a 
> 3-disk RAID1 to a 4-disk RAID6 seems to be via an intermediate 3-disk 
> RAID5, requiring two reshapes. I'd like to avoid one of them, if at all 
> possible.

Yes, not possible at present.
It might be as simple and finding the places that impose the limit and delete
them...

NeilBrown



> 
> Regards,
> Oliver
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Creating a 3-disk RAID6 array

[David Brown]

On 17/05/12 03:38, NeilBrown wrote:
> On Thu, 17 May 2012 03:24:26 +0200 Oliver Martin<oliver@volatilevoid.net>
> wrote:
>
>> Hi,
>>
>> is there any specific reason why md refuses to create a RAID6 array with
>> 3 disks? My (probably naive) understanding suggests it should be the
>> same as a 3-disk RAID1, similar to a 2-disk RAID5.
>
> I'll have to leave for for hpa to answer.  I've occasionally thought that
> maybe it should be fixed, but it never seemed worth the effort.
>
>>
>> The reason I'm asking is that I currently have space on three disks for
>> a new array, and would like to expand it when I add a fourth. I tried
>> this scenario with a few loopback devices, but the only way to go from a
>> 3-disk RAID1 to a 4-disk RAID6 seems to be via an intermediate 3-disk
>> RAID5, requiring two reshapes. I'd like to avoid one of them, if at all
>> possible.
>
> Yes, not possible at present.
> It might be as simple and finding the places that impose the limit and delete
> them...
>
> NeilBrown
>

Theoretically, a 3-disk RAID6 is like a 2-disk RAID5 or a 1-disk RAID1 - 
it is possible, but not much use except as a building block for 
expansion later.  mdadm supports 1-disk RAID1 (you need a "--force" to 
persuade mdadm that you know what you are doing, which is nice), and I 
think it supports a 2-disk RAID5, though I haven't tried it.  I can't 
think of any good reason for it /not/ to support 3-disk RAID6, as there 
is nothing in the algorithms to hinder it.

mvh.,

David

Re: Creating a 3-disk RAID6 array

[John Robinson]

On 17/05/2012 14:18, David Brown wrote:
[...]
> Theoretically, a 3-disk RAID6 is like a 2-disk RAID5 or a 1-disk RAID1 -
[...]
> I can't think of any good reason for it /not/ to support 3-disk
> RAID6, as there is nothing in the algorithms to hinder it.

I've a vague recollection of Peter Anvin saying the implementation is 
optimised in such a way that it won't work.

Cheers,

John.

Re: Creating a 3-disk RAID6 array

[David Brown]

On 17/05/2012 16:04, John Robinson wrote:
> On 17/05/2012 14:18, David Brown wrote:
> [...]
>> Theoretically, a 3-disk RAID6 is like a 2-disk RAID5 or a 1-disk RAID1 -
> [...]
>> I can't think of any good reason for it /not/ to support 3-disk
>> RAID6, as there is nothing in the algorithms to hinder it.
>
> I've a vague recollection of Peter Anvin saying the implementation is
> optimised in such a way that it won't work.
>

Okay, optimised implementation is a good reason for not supporting it! 
I only know the theory, I haven't studied the implementation well enough 
yet.  (One day, when I get time... :-)

> Cheers,
>
> John.
>

Re: Creating a 3-disk RAID6 array

[H. Peter Anvin]

On 05/17/2012 07:04 AM, John Robinson wrote:
> On 17/05/2012 14:18, David Brown wrote:
> [...]
>> Theoretically, a 3-disk RAID6 is like a 2-disk RAID5 or a 1-disk RAID1 -
> [...]
>> I can't think of any good reason for it /not/ to support 3-disk
>> RAID6, as there is nothing in the algorithms to hinder it.
> 
> I've a vague recollection of Peter Anvin saying the implementation is
> optimised in such a way that it won't work.
> 

Yes, it would have required introducing some odd special cases.  A
3-disk RAID-6 is bitwise identical to a 3-disk RAID-1, so if Neil wants
to he could add "instant reshaping" support in mdadm (add a disk to a
3-disk RAID-1 turning it into a 4-disk RAID-6; and similar for 2-disk
RAID-1 into 3-disk RAID-5.)

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Creating a 3-disk RAID6 array

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

On Fri, 18 May 2012 00:30:47 -0700 "H. Peter Anvin" <hpa@zytor.com> wrote:

> On 05/17/2012 07:04 AM, John Robinson wrote:
> > On 17/05/2012 14:18, David Brown wrote:
> > [...]
> >> Theoretically, a 3-disk RAID6 is like a 2-disk RAID5 or a 1-disk RAID1 -
> > [...]
> >> I can't think of any good reason for it /not/ to support 3-disk
> >> RAID6, as there is nothing in the algorithms to hinder it.
> > 
> > I've a vague recollection of Peter Anvin saying the implementation is
> > optimised in such a way that it won't work.
> > 
> 
> Yes, it would have required introducing some odd special cases.  A
> 3-disk RAID-6 is bitwise identical to a 3-disk RAID-1, so if Neil wants
> to he could add "instant reshaping" support in mdadm (add a disk to a
> 3-disk RAID-1 turning it into a 4-disk RAID-6; and similar for 2-disk
> RAID-1 into 3-disk RAID-5.)
> 
> 	-hpa
> 

Converting a 2-disk RAID-1 to a 3-disk RAID-5 is not instant.
You first convert a 2-disk RAID-1 to a 2-disk RAID-5.  Then you re-stripe the
RAID-5 to have 3-disks.  During this restripe, part of the array looks like a
2-disk RAID-5, and part looks like a 3-disk RAID-4.

To convert some sort of RAID1 to RAID6 we would need to be able to support a
3-disk RAID-6, if only for a relatively short time.

Possibly we could special case the parity-generation code to just copy the
data block when data-disks==1.  That sound easy enough...
But the raid456 module needs to understand this
3-way-RAID1-pretending-to-be-RAID6, we cannot use the raid1 module to do it.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Creating a 3-disk RAID6 array

[H. Peter Anvin]

On 05/18/2012 12:57 AM, NeilBrown wrote:
> 
> Possibly we could special case the parity-generation code to just
> copy the data block when data-disks==1.  That sound easy enough... 
> But the raid456 module needs to understand this 
> 3-way-RAID1-pretending-to-be-RAID6, we cannot use the raid1 module
> to do it.
> 

I actually think it will Just Work as-is... there might be some need
to modify the loop control logic.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Creating a 3-disk RAID6 array

[Dan Williams]

On Wed, May 16, 2012 at 6:38 PM, NeilBrown <neilb@suse.de> wrote:
> On Thu, 17 May 2012 03:24:26 +0200 Oliver Martin <oliver@volatilevoid.net>
> wrote:
>
>> Hi,
>>
>> is there any specific reason why md refuses to create a RAID6 array with
>> 3 disks? My (probably naive) understanding suggests it should be the
>> same as a 3-disk RAID1, similar to a 2-disk RAID5.
>
> I'll have to leave for for hpa to answer.  I've occasionally thought that
> maybe it should be fixed, but it never seemed worth the effort.

The math assumes 2 data disks.

>> The reason I'm asking is that I currently have space on three disks for
>> a new array, and would like to expand it when I add a fourth. I tried
>> this scenario with a few loopback devices, but the only way to go from a
>> 3-disk RAID1 to a 4-disk RAID6 seems to be via an intermediate 3-disk
>> RAID5, requiring two reshapes. I'd like to avoid one of them, if at all
>> possible.
>
> Yes, not possible at present.
> It might be as simple and finding the places that impose the limit and delete
> them...

You'd certainly need to route around the acceleration code, because
that increased the dependency on the assumption that there is always
two data disk slots.

--
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Creating a 3-disk RAID6 array

[Oliver Martin]

Am 17.05.2012 23:02, schrieb Dan Williams:
>> I'll have to leave for for hpa to answer.  I've occasionally thought that
>> maybe it should be fixed, but it never seemed worth the effort.
>
> The math assumes 2 data disks.

Aside from a BUG_ON in async_raid6_datap_recov, it seems like it should 
work based on a cursory glance. Especially the various gen_syndrome 
implementations and raid6_datap_recov look like they should produce the 
correct result.

>> Yes, not possible at present.
>> It might be as simple and finding the places that impose the limit and delete
>> them...
>
> You'd certainly need to route around the acceleration code, because
> that increased the dependency on the assumption that there is always
> two data disk slots.

This, however, makes it seem like lots of hassle for very little gain, 
given that the same on-disk data can much more cheaply be produced by 
using a RAID1. I think the more sensible thing to do is to add support 
for reshaping a RAID1 into a 4-disk RAID6. This should hopefully not be 
too different from the existing RAID5-RAID6 reshape, and is probably 
what I'll (at least try to) implement when the time comes to expand my 
array.

Oliver

Re: Creating a 3-disk RAID6 array

[Bill Davidsen]

Oliver Martin wrote:
> Hi,
>
> is there any specific reason why md refuses to create a RAID6 array with 3
> disks? My (probably naive) understanding suggests it should be the same as a
> 3-disk RAID1, similar to a 2-disk RAID5.
>
> The reason I'm asking is that I currently have space on three disks for a new
> array, and would like to expand it when I add a fourth. I tried this scenario
> with a few loopback devices, but the only way to go from a 3-disk RAID1 to a
> 4-disk RAID6 seems to be via an intermediate 3-disk RAID5, requiring two
> reshapes. I'd like to avoid one of them, if at all possible.
>
If you are definitely planning to add another drive fairly soon, you can create 
a four drive array with one missing. Given that the performance of a three way 
raid-6 is not going to be stellar anyway, since you write to every drive with 
every write, you actually might find it runs better only writing to two on some 
chunks.

It avoids the whole double reshape issue.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Creating a 3-disk RAID6 array

[Stan Hoeppner]

On 5/16/2012 8:24 PM, Oliver Martin wrote:
> Hi,
> 
> is there any specific reason why md refuses to create a RAID6 array with
> 3 disks? My (probably naive) understanding suggests it should be the
> same as a 3-disk RAID1, similar to a 2-disk RAID5.
> 
> The reason I'm asking is that I currently have space on three disks for
> a new array, and would like to expand it when I add a fourth. I tried
> this scenario with a few loopback devices, but the only way to go from a
> 3-disk RAID1 to a 4-disk RAID6 seems to be via an intermediate 3-disk
> RAID5, requiring two reshapes. I'd like to avoid one of them, if at all
> possible.

Make a 3 partition (seems you're using partitions) md RAID 10 (RAID 1E)
array with what you have now.  When you add a disk down the road, backup
your filesystem, create your 4 disk md RAID 6 array, format, restore.

Now for my $0.02:  You're better served all around with a 4 partition
RAID 10 or 1+0, than a 4 partition RAID 6.  Random write performance is
greatly superior, rebuild time is significantly lower, etc.  Sure it
would be nice if mdadm could reshape a 3 disk RAID10 into a 4 disk
RAID10, but it can't.  But since you should be doing a backup/restore
anyway, it doesn't matter.

-- 
Stan