Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

   A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?

   Occurred once on Linux 3.2.12, x86_64.

--

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
wrote:

>    A power outage has happened to system (which was in "suspend"
> state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
> fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
> auto-readonly mode with resync=DELAYED. When they flipped to
> Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?

Maybe.  But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Did that arrays still have their bitmaps?
Where they degraded?
How many bits were set in the bitmaps (probably too late to check now)?

Is it reproducable ?  ... OK, I don't really expect you to try to reproduce
it, but if you could that would be awesome!

NeilBrown


> 
>    Occurred once on Linux 3.2.12, x86_64.
> 
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 27 May 2012 20:00, NeilBrown <neilb@suse.de> wrote:
> On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> wrote:
>>    A power outage has happened to system (which was in "suspend"
>> state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
>> fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
>> auto-readonly mode with resync=DELAYED. When they flipped to
>> Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
>
> Maybe.  But with so little detail it is hard to say.
> What made you think the bitmaps weren't being used?

   Well, it is reasonable to assume that host being put in suspend
mode wouldn't have much not synced data at all. RAID-10 had been
progressing its resync quite fast, it was obvious that it had been
using bitmaps for that. Moreover — RAID-10 was the most I/O active
comparing to others, AFAIR.

> Did that arrays still have their bitmaps?

   I still have that RAID-6, yep:

md127 : active raid6 sdg5[2] sdf5[6] sde5[7] sda5[0] sdd5[3] sdc5[1]
      1071984640 blocks super 1.0 level 6, 1024k chunk, algorithm 2
[6/6] [UUUUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

> Where they degraded?

   No way.

> How many bits were set in the bitmaps (probably too late to check now)?

   Please see above. And the RAID-5 one was recently dismissed, alas.

> Is it reproducable ?  ... OK, I don't really expect you to try to reproduce
> it, but if you could that would be awesome!

   Well, might be VirtualBox would allow for such kind of experiment, dunno. )

   P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
bug causing kernel panic when using WIB -- couldn't you please point
out is it due to they're missing some important bugfix you made later?

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2378 bytes --]

On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
wrote:

> On 27 May 2012 20:00, NeilBrown <neilb@suse.de> wrote:
> > On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> > wrote:
> >>    A power outage has happened to system (which was in "suspend"
> >> state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
> >> fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
> >> auto-readonly mode with resync=DELAYED. When they flipped to
> >> Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
> >
> > Maybe.  But with so little detail it is hard to say.
> > What made you think the bitmaps weren't being used?
> 
>    Well, it is reasonable to assume that host being put in suspend
> mode wouldn't have much not synced data at all. RAID-10 had been
> progressing its resync quite fast, it was obvious that it had been
> using bitmaps for that. Moreover — RAID-10 was the most I/O active
> comparing to others, AFAIR.

Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
help in analysing a situation to find any bugs.
I can only really work with concrete facts.  Without them I cannot help.

> 
> > Did that arrays still have their bitmaps?
> 
>    I still have that RAID-6, yep:
> 
> md127 : active raid6 sdg5[2] sdf5[6] sde5[7] sda5[0] sdd5[3] sdc5[1]
>       1071984640 blocks super 1.0 level 6, 1024k chunk, algorithm 2
> [6/6] [UUUUUU]
>       bitmap: 0/1 pages [0KB], 131072KB chunk
> 
> > Where they degraded?
> 
>    No way.
> 
> > How many bits were set in the bitmaps (probably too late to check now)?
> 
>    Please see above. And the RAID-5 one was recently dismissed, alas.
> 
> > Is it reproducable ?  ... OK, I don't really expect you to try to reproduce
> > it, but if you could that would be awesome!
> 
>    Well, might be VirtualBox would allow for such kind of experiment, dunno. )
> 
>    P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
> bug causing kernel panic when using WIB -- couldn't you please point
> out is it due to they're missing some important bugfix you made later?

I have no time or interest for submiting bug reports to distros that I don't
use.  If the presence of the bug concerns you, I suggest you report it.

NeilBrown


> 
> --


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 28 May 2012 09:45, NeilBrown <neilb@suse.de> wrote:
> On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> wrote:
>
>> On 27 May 2012 20:00, NeilBrown <neilb@suse.de> wrote:
>> > On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
>> > wrote:
>> >>    A power outage has happened to system (which was in "suspend"
>> >> state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
>> >> fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
>> >> auto-readonly mode with resync=DELAYED. When they flipped to
>> >> Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
>> >
>> > Maybe.  But with so little detail it is hard to say.
>> > What made you think the bitmaps weren't being used?
>>
>>    Well, it is reasonable to assume that host being put in suspend
>> mode wouldn't have much not synced data at all. RAID-10 had been
>> progressing its resync quite fast, it was obvious that it had been
>> using bitmaps for that. Moreover — RAID-10 was the most I/O active
>> comparing to others, AFAIR.
>
> Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
> help in analysing a situation to find any bugs.
> I can only really work with concrete facts.  Without them I cannot help.

   Well, than probably you could tell me how one does find out whether
recovery uses WIB, or it doesn't? :-) I mean aside from vague
estimating based on watching re-sync progressing and considering
overall time it finally took to complete.

>>    P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
>> bug causing kernel panic when using WIB -- couldn't you please point
>> out is it due to they're missing some important bugfix you made later?
>
> I have no time or interest for submiting bug reports to distros that I don't
> use.  If the presence of the bug concerns you, I suggest you report it.

   I thought you could have some recollections bout this rather nasty
behavior. It has nothing to do with bug-reports for RedHat since
vanilla 2.6.18 for sure has the same bug, and hopefully it has been
fixed in later releases -- that was the only thing I was asking you
about. But your competent suggestions are also appreciated, of course.

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3076 bytes --]

On Mon, 28 May 2012 10:37:14 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
wrote:

> On 28 May 2012 09:45, NeilBrown <neilb@suse.de> wrote:
> > On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> > wrote:
> >
> >> On 27 May 2012 20:00, NeilBrown <neilb@suse.de> wrote:
> >> > On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> >> > wrote:
> >> >>    A power outage has happened to system (which was in "suspend"
> >> >> state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
> >> >> fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
> >> >> auto-readonly mode with resync=DELAYED. When they flipped to
> >> >> Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
> >> >
> >> > Maybe.  But with so little detail it is hard to say.
> >> > What made you think the bitmaps weren't being used?
> >>
> >>    Well, it is reasonable to assume that host being put in suspend
> >> mode wouldn't have much not synced data at all. RAID-10 had been
> >> progressing its resync quite fast, it was obvious that it had been
> >> using bitmaps for that. Moreover — RAID-10 was the most I/O active
> >> comparing to others, AFAIR.
> >
> > Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
> > help in analysing a situation to find any bugs.
> > I can only really work with concrete facts.  Without them I cannot help.
> 
>    Well, than probably you could tell me how one does find out whether
> recovery uses WIB, or it doesn't? :-) I mean aside from vague
> estimating based on watching re-sync progressing and considering
> overall time it finally took to complete.

The best indicator is total time that it takes (which can probably be
extracted from logs as start and end are logged).  Divide that into size of a
device to get average MB/sec.  If the bitmap was used, that will normally be
much less the best throughput of the device.

> 
> >>    P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
> >> bug causing kernel panic when using WIB -- couldn't you please point
> >> out is it due to they're missing some important bugfix you made later?
> >
> > I have no time or interest for submiting bug reports to distros that I don't
> > use.  If the presence of the bug concerns you, I suggest you report it.
> 
>    I thought you could have some recollections bout this rather nasty
> behavior. It has nothing to do with bug-reports for RedHat since
> vanilla 2.6.18 for sure has the same bug, and hopefully it has been
> fixed in later releases -- that was the only thing I was asking you
> about. But your competent suggestions are also appreciated, of course.
> 

2.6.18 is a long time ago.  I don't really have any recollections - so many
bugs, so many releases.  I'm not even entirely sure what bug you are
referring to.  I would probably have tagged any patch for it to be included
in -stable, but I cannot be certain without looking more deeply.

NeilBrown

> --


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 28 May 2012 10:54, NeilBrown <neilb@suse.de> wrote:
[...]
> The best indicator is total time that it takes (which can probably be
> extracted from logs as start and end are logged).  Divide that into size of a
> device to get average MB/sec.  If the bitmap was used, that will normally be
> much less the best throughput of the device.

   That's exactly why I am stating it didn't use WIBs 2 times of 3. It
went resyncing from beginning till the end -- I had my finger on its
pulse. :)

   I can't promise (due to popular "lack of time" disease, yeah), but
if I'll get on it using VirtualBox or real environment [Lord forbid
:)] once again, I'll let you know, sure.

[...]
> 2.6.18 is a long time ago.  I don't really have any recollections - so many
> bugs, so many releases.  I'm not even entirely sure what bug you are
> referring to.  I would probably have tagged any patch for it to be included
> in -stable, but I cannot be certain without looking more deeply.

   Okay, thanks, it gonna take some kind of bisecting I guess, then.

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 28 May 2012 12:03, Igor M Podlesny <for.poige+lsr@gmail.com> wrote:
> On 28 May 2012 10:54, NeilBrown <neilb@suse.de> wrote:
> [...]
>> The best indicator is total time that it takes (which can probably be
>> extracted from logs as start and end are logged).  Divide that into size of a
>> device to get average MB/sec.  If the bitmap was used, that will normally be
>> much less the best throughput of the device.
>
>   That's exactly why I am stating it didn't use WIBs 2 times of 3. It
> went resyncing from beginning till the end -- I had my finger on its
> pulse. :)
>
>   I can't promise (due to popular "lack of time" disease, yeah), but
> if I'll get on it using VirtualBox or real environment [Lord forbid
> :)] once again, I'll let you know, sure.

   Here it is: http://pastie.org/4118133

   — 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
Reboot, resync, no bitmaps use.

   And, BTW, it's rather slow in despite of having max. stripe_cache_size set.

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1783 bytes --]

On Wed, 20 Jun 2012 10:49:32 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
wrote:

> On 28 May 2012 12:03, Igor M Podlesny <for.poige+lsr@gmail.com> wrote:
> > On 28 May 2012 10:54, NeilBrown <neilb@suse.de> wrote:
> > [...]
> >> The best indicator is total time that it takes (which can probably be
> >> extracted from logs as start and end are logged).  Divide that into size of a
> >> device to get average MB/sec.  If the bitmap was used, that will normally be
> >> much less the best throughput of the device.
> >
> >   That's exactly why I am stating it didn't use WIBs 2 times of 3. It
> > went resyncing from beginning till the end -- I had my finger on its
> > pulse. :)
> >
> >   I can't promise (due to popular "lack of time" disease, yeah), but
> > if I'll get on it using VirtualBox or real environment [Lord forbid
> > :)] once again, I'll let you know, sure.
> 
>    Here it is: http://pastie.org/4118133

This doesn't appear to show the "start to end" that I suggested, just 10% to
30%, but it does suggest that the bitmap isn't speeding things up at all, at
least for that part of the array... That is assuming you are talking about
md124.  I note that md125 doesn't have a bitmap.  Is that what you are
referring to?

The next thing to do would be to look at the bitmap immediately after reboot
to see how many bits are set.  Maybe lots are set for some reason.

> 
>    — 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
> Reboot, resync, no bitmaps use.
> 
>    And, BTW, it's rather slow in despite of having max. stripe_cache_size set.
> 

A larger stripe cache isn't going to affect resync speed much (I think).
50MB/sec on each drive seems fairly good to me.  What were you expecting?

NeilBrown



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 26 June 2012 14:18, NeilBrown <neilb@suse.de> wrote:
> On Wed, 20 Jun 2012 10:49:32 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
> wrote:
[…]
>>    Here it is: http://pastie.org/4118133
>
> This doesn't appear to show the "start to end" that I suggested, just 10% to
> 30%, but it does suggest that the bitmap isn't speeding things up at all, at
> least for that part of the array... That is assuming you are talking about

   Well, I decided that's enough to be "pasted", but anyway, I can
assure you — it took ~ 1 hour, as it was estimated, so yeah — no use
of bitmap. For sure.

> md124.  I note that md125 doesn't have a bitmap.  Is that what you are
> referring to?

   Yeah, md125 is legacy RAID which is not being in use for a long
time, it's just hanging around till re-layout happens.
>
> The next thing to do would be to look at the bitmap immediately after reboot
> to see how many bits are set.  Maybe lots are set for some reason.
>
>>
>>    — 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
>> Reboot, resync, no bitmaps use.
>>
>>    And, BTW, it's rather slow in despite of having max. stripe_cache_size set.
>>
>
> A larger stripe cache isn't going to affect resync speed much (I think).
> 50MB/sec on each drive seems fairly good to me.  What were you expecting?

   I see, thanks.

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi! Strange issue with LSR -- bitmaps hadn't been used during 2 of 3 RAIDs resync

[Igor M Podlesny]

On 26 June 2012 16:55, Igor M Podlesny <for.poige+lsr@gmail.com> wrote:
> On 26 June 2012 14:18, NeilBrown <neilb@suse.de> wrote:
>> On Wed, 20 Jun 2012 10:49:32 +0800 Igor M Podlesny <for.poige+lsr@gmail.com>
>> wrote:
> […]
>>>    Here it is: http://pastie.org/4118133
>>
>> This doesn't appear to show the "start to end" that I suggested, just 10% to
>> 30%, but it does suggest that the bitmap isn't speeding things up at all, at
>> least for that part of the array... That is assuming you are talking about
>
>   Well, I decided that's enough to be "pasted", but anyway, I can
> assure you — it took ~ 1 hour, as it was estimated, so yeah — no use
> of bitmap. For sure.
>
>> md124.  I note that md125 doesn't have a bitmap.  Is that what you are
>> referring to?
>
>   Yeah, md125 is legacy RAID which is not being in use for a long
> time, it's just hanging around till re-layout happens.
>>
>> The next thing to do would be to look at the bitmap immediately after reboot
>> to see how many bits are set.  Maybe lots are set for some reason.

   That would more than unlikely due to:
   — it's prepended with several sync-sleeps
   — there's no intensive I/O
   — kernel should have flush all buffers on suspend as well

--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html