about raid5 recovery when created

about raid5 recovery when created

[hank peng]

Hi, all:
As we know, when a raid5 array is created, recovery will be going on
which involves some read, one xor and one write. Since there is no
real data in the disk at the time, besides, if I am willing to wait
for recovery to complete and then use this raid5, how about adding
support for a fast recovery method? Right now, what is in my mind is
zero all disks which belong to this raid5. I think it will increase
raid5 recovery speed when created and decrease CPU usage, since all
zero is also XORed.
What do raid developers think?

-- 
The simplest is not all best but the best is surely the simplest!

Re: about raid5 recovery when created

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1197 bytes --]

On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:

> Hi, all:
> As we know, when a raid5 array is created, recovery will be going on
> which involves some read, one xor and one write. Since there is no
> real data in the disk at the time, besides, if I am willing to wait
> for recovery to complete and then use this raid5, how about adding
> support for a fast recovery method? Right now, what is in my mind is
> zero all disks which belong to this raid5. I think it will increase
> raid5 recovery speed when created and decrease CPU usage, since all
> zero is also XORed.
> What do raid developers think?
> 
It'll decrease CPU usage but increase I/O - you're now needing to write
to all disks.  Most systems will be I/O limited rather than CPU limited,
so the current approach works better.  If you want to zero the disks
then do this before creating the array - you can then use --assume-clean
to skip the resync process.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: about raid5 recovery when created

[hank peng]

2009/12/8 Robin Hill <robin@robinhill.me.uk>:
> On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>
>> Hi, all:
>> As we know, when a raid5 array is created, recovery will be going on
>> which involves some read, one xor and one write. Since there is no
>> real data in the disk at the time, besides, if I am willing to wait
>> for recovery to complete and then use this raid5, how about adding
>> support for a fast recovery method? Right now, what is in my mind is
>> zero all disks which belong to this raid5. I think it will increase
>> raid5 recovery speed when created and decrease CPU usage, since all
>> zero is also XORed.
>> What do raid developers think?
>>
> It'll decrease CPU usage but increase I/O - you're now needing to write
> to all disks.  Most systems will be I/O limited rather than CPU limited,
> so the current approach works better.  If you want to zero the disks
> then do this before creating the array - you can then use --assume-clean
> to skip the resync process.
>
I think --assume-clean is used mostly when doing performance test and
can't be used when creating a raid5 array using new disk, because
later read and write operation make assumption that all stripe is
XORed. Correct me if I am wrong.

> Cheers,
>    Robin
> --
>     ___
>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>   / / )      | Little Jim says ....                            |
>  // !!       |      "He fallen in de water !!"                 |
>



-- 
The simplest is not all best but the best is surely the simplest!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

On Tue Dec 08, 2009 at 09:49:48PM +0800, hank peng wrote:

> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
> > On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
> >
> >> Hi, all:
> >> As we know, when a raid5 array is created, recovery will be going on
> >> which involves some read, one xor and one write. Since there is no
> >> real data in the disk at the time, besides, if I am willing to wait
> >> for recovery to complete and then use this raid5, how about adding
> >> support for a fast recovery method? Right now, what is in my mind is
> >> zero all disks which belong to this raid5. I think it will increase
> >> raid5 recovery speed when created and decrease CPU usage, since all
> >> zero is also XORed.
> >> What do raid developers think?
> >>
> > It'll decrease CPU usage but increase I/O - you're now needing to write
> > to all disks.  Most systems will be I/O limited rather than CPU limited,
> > so the current approach works better.  If you want to zero the disks
> > then do this before creating the array - you can then use --assume-clean
> > to skip the resync process.
> >
> I think --assume-clean is used mostly when doing performance test and
> can't be used when creating a raid5 array using new disk, because
> later read and write operation make assumption that all stripe is
> XORed. Correct me if I am wrong.
> 
You're correct - that's why I said to zero all the disks first so the
XOR data is all correct.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: about raid5 recovery when created

[hank peng]

2009/12/8 Robin Hill <robin@robinhill.me.uk>:
> On Tue Dec 08, 2009 at 09:49:48PM +0800, hank peng wrote:
>
>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>> > On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>> >
>> >> Hi, all:
>> >> As we know, when a raid5 array is created, recovery will be going on
>> >> which involves some read, one xor and one write. Since there is no
>> >> real data in the disk at the time, besides, if I am willing to wait
>> >> for recovery to complete and then use this raid5, how about adding
>> >> support for a fast recovery method? Right now, what is in my mind is
>> >> zero all disks which belong to this raid5. I think it will increase
>> >> raid5 recovery speed when created and decrease CPU usage, since all
>> >> zero is also XORed.
>> >> What do raid developers think?
>> >>
>> > It'll decrease CPU usage but increase I/O - you're now needing to write
>> > to all disks.  Most systems will be I/O limited rather than CPU limited,
>> > so the current approach works better.  If you want to zero the disks
>> > then do this before creating the array - you can then use --assume-clean
>> > to skip the resync process.
>> >
>> I think --assume-clean is used mostly when doing performance test and
>> can't be used when creating a raid5 array using new disk, because
>> later read and write operation make assumption that all stripe is
>> XORed. Correct me if I am wrong.
>>
> You're correct - that's why I said to zero all the disks first so the
> XOR data is all correct.
>
I think this function is better to be implemented in kernel raid
layer, not in user space(for example using dd command).
In this way, we can get good performance and lower cpu usage, also, we
can make this function be part of raid code so that it can be managed
by mdadm
> Cheers,
>    Robin
> --
>     ___
>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>   / / )      | Little Jim says ....                            |
>  // !!       |      "He fallen in de water !!"                 |
>



-- 
The simplest is not all best but the best is surely the simplest!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[Michael Evans]

On Tue, Dec 8, 2009 at 6:03 AM, hank peng <pengxihan@gmail.com> wrote:
> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>> On Tue Dec 08, 2009 at 09:49:48PM +0800, hank peng wrote:
>>
>>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>>> > On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>>> >
>>> >> Hi, all:
>>> >> As we know, when a raid5 array is created, recovery will be going on
>>> >> which involves some read, one xor and one write. Since there is no
>>> >> real data in the disk at the time, besides, if I am willing to wait
>>> >> for recovery to complete and then use this raid5, how about adding
>>> >> support for a fast recovery method? Right now, what is in my mind is
>>> >> zero all disks which belong to this raid5. I think it will increase
>>> >> raid5 recovery speed when created and decrease CPU usage, since all
>>> >> zero is also XORed.
>>> >> What do raid developers think?
>>> >>
>>> > It'll decrease CPU usage but increase I/O - you're now needing to write
>>> > to all disks.  Most systems will be I/O limited rather than CPU limited,
>>> > so the current approach works better.  If you want to zero the disks
>>> > then do this before creating the array - you can then use --assume-clean
>>> > to skip the resync process.
>>> >
>>> I think --assume-clean is used mostly when doing performance test and
>>> can't be used when creating a raid5 array using new disk, because
>>> later read and write operation make assumption that all stripe is
>>> XORed. Correct me if I am wrong.
>>>
>> You're correct - that's why I said to zero all the disks first so the
>> XOR data is all correct.
>>
> I think this function is better to be implemented in kernel raid
> layer, not in user space(for example using dd command).
> In this way, we can get good performance and lower cpu usage, also, we
> can make this function be part of raid code so that it can be managed
> by mdadm
>> Cheers,
>>    Robin
>> --
>>     ___
>>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>>   / / )      | Little Jim says ....                            |
>>  // !!       |      "He fallen in de water !!"                 |
>>
>
>
>
> --
> The simplest is not all best but the best is surely the simplest!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

How about documenting this better?  'zeroing all underlying devices
then creating with --assume-clean' will be clean because the parity
algorithm is even (or similar to 'even parity')?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[hank peng]

2009/12/9 Michael Evans <mjevans1983@gmail.com>:
> On Tue, Dec 8, 2009 at 6:03 AM, hank peng <pengxihan@gmail.com> wrote:
>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>>> On Tue Dec 08, 2009 at 09:49:48PM +0800, hank peng wrote:
>>>
>>>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>>>> > On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>>>> >
>>>> >> Hi, all:
>>>> >> As we know, when a raid5 array is created, recovery will be going on
>>>> >> which involves some read, one xor and one write. Since there is no
>>>> >> real data in the disk at the time, besides, if I am willing to wait
>>>> >> for recovery to complete and then use this raid5, how about adding
>>>> >> support for a fast recovery method? Right now, what is in my mind is
>>>> >> zero all disks which belong to this raid5. I think it will increase
>>>> >> raid5 recovery speed when created and decrease CPU usage, since all
>>>> >> zero is also XORed.
>>>> >> What do raid developers think?
>>>> >>
>>>> > It'll decrease CPU usage but increase I/O - you're now needing to write
>>>> > to all disks.  Most systems will be I/O limited rather than CPU limited,
>>>> > so the current approach works better.  If you want to zero the disks
>>>> > then do this before creating the array - you can then use --assume-clean
>>>> > to skip the resync process.
>>>> >
>>>> I think --assume-clean is used mostly when doing performance test and
>>>> can't be used when creating a raid5 array using new disk, because
>>>> later read and write operation make assumption that all stripe is
>>>> XORed. Correct me if I am wrong.
>>>>
>>> You're correct - that's why I said to zero all the disks first so the
>>> XOR data is all correct.
>>>
>> I think this function is better to be implemented in kernel raid
>> layer, not in user space(for example using dd command).
>> In this way, we can get good performance and lower cpu usage, also, we
>> can make this function be part of raid code so that it can be managed
>> by mdadm
>>> Cheers,
>>>    Robin
>>> --
>>>     ___
>>>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>>>   / / )      | Little Jim says ....                            |
>>>  // !!       |      "He fallen in de water !!"                 |
>>>
>>
>>
>>
>> --
>> The simplest is not all best but the best is surely the simplest!
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> How about documenting this better?  'zeroing all underlying devices
> then creating with --assume-clean' will be clean because the parity
> algorithm is even (or similar to 'even parity')?
> --
Yes, that's what I am thinking.
I think it is better to implement this function in kernel's md layer.
I wonder what Neil Brown think of this?

> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
The simplest is not all best but the best is surely the simplest!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[Neil Brown]

On Wed, 9 Dec 2009 19:29:04 +0800
hank peng <pengxihan@gmail.com> wrote:

> I think it is better to implement this function in kernel's md layer.
> I wonder what Neil Brown think of this?

I don't think it is worth the effort.
You probably would save some CPU time as you don't need to XOR, but as has
been said, we are usually IO bound, not CPU bound.

With the current arrangement, you can start using the array immediately - you
don't have to wait for the initial recovery to complete.
If you zeroed all devices at create time, you would have to wait for that to
complete before using the array.

So I see very little gain, and significant cost.

NeilBrown

Re: about raid5 recovery when created

[Michael Evans]

On Wed, Dec 9, 2009 at 5:43 PM, Neil Brown <neilb@suse.de> wrote:
> On Wed, 9 Dec 2009 19:29:04 +0800
> hank peng <pengxihan@gmail.com> wrote:
>
>> I think it is better to implement this function in kernel's md layer.
>> I wonder what Neil Brown think of this?
>
> I don't think it is worth the effort.
> You probably would save some CPU time as you don't need to XOR, but as has
> been said, we are usually IO bound, not CPU bound.
>
> With the current arrangement, you can start using the array immediately - you
> don't have to wait for the initial recovery to complete.
> If you zeroed all devices at create time, you would have to wait for that to
> complete before using the array.
>
> So I see very little gain, and significant cost.
>
> NeilBrown
>
>

When I assemble an array I tend to have checked the devices before
hand; it would not be difficult to make the final pass a zeroing pass
if I knew I could vastly speed up post-assembly performance.  As I
stated, it's merely a lack of clarity in the documentation.

Re: about raid5 recovery when created

[Neil Brown]

On Wed, 9 Dec 2009 19:34:04 -0800
Michael Evans <mjevans1983@gmail.com> wrote:

> On Wed, Dec 9, 2009 at 5:43 PM, Neil Brown <neilb@suse.de> wrote:
> > On Wed, 9 Dec 2009 19:29:04 +0800
> > hank peng <pengxihan@gmail.com> wrote:
> >
> >> I think it is better to implement this function in kernel's md layer.
> >> I wonder what Neil Brown think of this?
> >
> > I don't think it is worth the effort.
> > You probably would save some CPU time as you don't need to XOR, but as has
> > been said, we are usually IO bound, not CPU bound.
> >
> > With the current arrangement, you can start using the array immediately - you
> > don't have to wait for the initial recovery to complete.
> > If you zeroed all devices at create time, you would have to wait for that to
> > complete before using the array.
> >
> > So I see very little gain, and significant cost.
> >
> > NeilBrown
> >
> >
> 
> When I assemble an array I tend to have checked the devices before
> hand; it would not be difficult to make the final pass a zeroing pass
> if I knew I could vastly speed up post-assembly performance.  As I
> stated, it's merely a lack of clarity in the documentation.

If you would like to create a patch against the man page, I would be happy to
accept it.

NeilBrown

Re: about raid5 recovery when created

[Neil Brown]

Thanks.  I have applied this patch.

NeilBrown



On Wed, 9 Dec 2009 21:52:18 -0800 (PST)
mjevans1983@gmail.com wrote:

> On Wed, Dec 9, 2009 at 7:59 PM, Neil Brown <neilb@suse.de> wrote:
> > On Wed, 9 Dec 2009 19:34:04 -0800
> > Michael Evans <mjevans1983@gmail.com> wrote:
> >
> >> On Wed, Dec 9, 2009 at 5:43 PM, Neil Brown <neilb@suse.de> wrote:
> >> > On Wed, 9 Dec 2009 19:29:04 +0800
> >> > hank peng <pengxihan@gmail.com> wrote:
> >> >
> >> >> I think it is better to implement this function in kernel's md layer.
> >> >> I wonder what Neil Brown think of this?
> >> >
> >> > I don't think it is worth the effort.
> >> > You probably would save some CPU time as you don't need to XOR, but as has
> >> > been said, we are usually IO bound, not CPU bound.
> >> >
> >> > With the current arrangement, you can start using the array immediately - you
> >> > don't have to wait for the initial recovery to complete.
> >> > If you zeroed all devices at create time, you would have to wait for that to
> >> > complete before using the array.
> >> >
> >> > So I see very little gain, and significant cost.
> >> >
> >> > NeilBrown
> >> >
> >> >
> >>
> >> When I assemble an array I tend to have checked the devices before
> >> hand; it would not be difficult to make the final pass a zeroing pass
> >> if I knew I could vastly speed up post-assembly performance.  As I
> >> stated, it's merely a lack of clarity in the documentation.
> >
> > If you would like to create a patch against the man page, I would be happy to
> > accept it.
> >
> > NeilBrown
> >
> 
> diff --git a/mdadm.8 b/mdadm.8
> index f8f240a..29e098b 100644
> --- a/mdadm.8
> +++ b/mdadm.8
> @@ -648,6 +648,12 @@ data will be affected unless you actually write to the array.  It can
>  also be used when creating a RAID1 or RAID10 if you want to avoid the
>  initial resync, however this practice \(em while normally safe \(em is not
>  recommended.  Use this only if you really know what you are doing.
> +.IP
> +When the devices that will be part of a new array were filled
> +with zeros before creation the operator knows the array is
> +actually clean. If that is the case, such as after running
> +badblocks, this argument can be used to tell mdadm the
> +facts the operator knows.
> 
>  .TP
>  .BR \-\-backup\-file=

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[Michael Evans]

On Wed, Dec 9, 2009 at 3:30 AM, hank peng <pengxihan@gmail.com> wrote:
> 2009/12/9 Michael Evans <mjevans1983@gmail.com>:
>> On Tue, Dec 8, 2009 at 6:03 AM, hank peng <pengxihan@gmail.com> wrote:
>>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>>>> On Tue Dec 08, 2009 at 09:49:48PM +0800, hank peng wrote:
>>>>
>>>>> 2009/12/8 Robin Hill <robin@robinhill.me.uk>:
>>>>> > On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>>>>> >
>>>>> >> Hi, all:
>>>>> >> As we know, when a raid5 array is created, recovery will be going on
>>>>> >> which involves some read, one xor and one write. Since there is no
>>>>> >> real data in the disk at the time, besides, if I am willing to wait
>>>>> >> for recovery to complete and then use this raid5, how about adding
>>>>> >> support for a fast recovery method? Right now, what is in my mind is
>>>>> >> zero all disks which belong to this raid5. I think it will increase
>>>>> >> raid5 recovery speed when created and decrease CPU usage, since all
>>>>> >> zero is also XORed.
>>>>> >> What do raid developers think?
>>>>> >>
>>>>> > It'll decrease CPU usage but increase I/O - you're now needing to write
>>>>> > to all disks.  Most systems will be I/O limited rather than CPU limited,
>>>>> > so the current approach works better.  If you want to zero the disks
>>>>> > then do this before creating the array - you can then use --assume-clean
>>>>> > to skip the resync process.
>>>>> >
>>>>> I think --assume-clean is used mostly when doing performance test and
>>>>> can't be used when creating a raid5 array using new disk, because
>>>>> later read and write operation make assumption that all stripe is
>>>>> XORed. Correct me if I am wrong.
>>>>>
>>>> You're correct - that's why I said to zero all the disks first so the
>>>> XOR data is all correct.
>>>>
>>> I think this function is better to be implemented in kernel raid
>>> layer, not in user space(for example using dd command).
>>> In this way, we can get good performance and lower cpu usage, also, we
>>> can make this function be part of raid code so that it can be managed
>>> by mdadm
>>>> Cheers,
>>>>    Robin
>>>> --
>>>>     ___
>>>>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>>>>   / / )      | Little Jim says ....                            |
>>>>  // !!       |      "He fallen in de water !!"                 |
>>>>
>>>
>>>
>>>
>>> --
>>> The simplest is not all best but the best is surely the simplest!
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> How about documenting this better?  'zeroing all underlying devices
>'Lost in translation'
>> then creating with --assume-clean' will be clean because the parity
>> algorithm is even (or similar to 'even parity')?
>> --

Maybe this will translate more easily.

The documentation should be more explicit.  "When the devices the RAID
is made of are filled with zero's before RAID creation --assume-clean
can be used because the parity will already be correct."
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: about raid5 recovery when created

[hank peng]

2009/12/8 Robin Hill <robin@robinhill.me.uk>:
> On Tue Dec 08, 2009 at 09:01:23PM +0800, hank peng wrote:
>
>> Hi, all:
>> As we know, when a raid5 array is created, recovery will be going on
>> which involves some read, one xor and one write. Since there is no
>> real data in the disk at the time, besides, if I am willing to wait
>> for recovery to complete and then use this raid5, how about adding
>> support for a fast recovery method? Right now, what is in my mind is
>> zero all disks which belong to this raid5. I think it will increase
>> raid5 recovery speed when created and decrease CPU usage, since all
>> zero is also XORed.
>> What do raid developers think?
>>
> It'll decrease CPU usage but increase I/O - you're now needing to write
> to all disks.  Most systems will be I/O limited rather than CPU limited,
My CPU is an embedded arch, 1.3GHz, single core, so CPU usage is important.
In low-end NAS box, I think, this requirment is normal.

> so the current approach works better.  If you want to zero the disks
> then do this before creating the array - you can then use --assume-clean
> to skip the resync process.
>
> Cheers,
>    Robin
> --
>     ___
>    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
>   / / )      | Little Jim says ....                            |
>  // !!       |      "He fallen in de water !!"                 |
>



-- 
The simplest is not all best but the best is surely the simplest!
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html