Need clarification on raid1 resync behavior with bitmap support

Need clarification on raid1 resync behavior with bitmap support

[Mike Snitzer]

On 6/1/06, NeilBrown <neilb@suse.de> wrote:
>
> When an array has a bitmap, a device can be removed and re-added
> and only blocks changes since the removal (as recorded in the bitmap)
> will be resynced.

Neil,

Does the same apply when a bitmap-enabled raid1's member goes faulty?
Meaning even if a member is faulty, when the user removes and re-adds
the faulty device the raid1 rebuild _should_ leverage the bitmap
during a resync right?

I've seen messages like:
[12068875.690255] raid1: raid set md0 active with 2 out of 2 mirrors
[12068875.690284] md0: bitmap file is out of date (0 < 1) -- forcing
full recovery
[12068875.690289] md0: bitmap file is out of date, doing full recovery
[12068875.710214] md0: bitmap initialized from disk: read 5/5 pages,
set 131056 bits, status: 0
[12068875.710222] created bitmap (64 pages) for device md0

Could you share the other situations where a bitmap-enabled raid1
_must_ perform a full recovery?
- Correct me if I'm wrong, but one that comes to mind is when a server
reboots (after cleanly stopping a raid1 array that had a faulty
member) and then either:
1) assembles the array with the previously faulty member now available

2) assembles the array with the same faulty member missing.  The user
later re-adds the faulty member

AFAIK both scenarios would bring about a full resync.

regards,
Mike

Re: Need clarification on raid1 resync behavior with bitmap support

[Neil Brown]

On Saturday July 21, snitzer@gmail.com wrote:
> On 6/1/06, NeilBrown <neilb@suse.de> wrote:
> >
> > When an array has a bitmap, a device can be removed and re-added
> > and only blocks changes since the removal (as recorded in the bitmap)
> > will be resynced.
> 
> Neil,
> 
> Does the same apply when a bitmap-enabled raid1's member goes faulty?
> Meaning even if a member is faulty, when the user removes and re-adds
> the faulty device the raid1 rebuild _should_ leverage the bitmap
> during a resync right?

Yes and no.

While the array is degraded, bits are never cleared from the bitmap.
So if you remove and re-add a device from a degraded array, then all
the block that have changed since the array became degraded will need
to be recovered.  This will likely be more than where changed while
the device was temporarily removed from the array, but probably much
less than the entire array.

> 
> I've seen messages like:
> [12068875.690255] raid1: raid set md0 active with 2 out of 2 mirrors
> [12068875.690284] md0: bitmap file is out of date (0 < 1) -- forcing
> full recovery
> [12068875.690289] md0: bitmap file is out of date, doing full recovery
> [12068875.710214] md0: bitmap initialized from disk: read 5/5 pages,
> set 131056 bits, status: 0
> [12068875.710222] created bitmap (64 pages) for device md0
> 
> Could you share the other situations where a bitmap-enabled raid1
> _must_ perform a full recovery?

When you add a new drive.  When you create a new bitmap.  I think that
should be all.

> - Correct me if I'm wrong, but one that comes to mind is when a server
> reboots (after cleanly stopping a raid1 array that had a faulty
> member) and then either:
> 1) assembles the array with the previously faulty member now
> available
> 
> 2) assembles the array with the same faulty member missing.  The user
> later re-adds the faulty member
> 
> AFAIK both scenarios would bring about a full resync.

Only if the drive is not recognised as the original member.
Can you test this out and report a sequence of events that causes a
full resync?

NeilBrown

Re: Need clarification on raid1 resync behavior with bitmap support

[Mike Snitzer]

On 7/23/07, Neil Brown <neilb@suse.de> wrote:
> On Saturday July 21, snitzer@gmail.com wrote:

> > Could you share the other situations where a bitmap-enabled raid1
> > _must_ perform a full recovery?
>
> When you add a new drive.  When you create a new bitmap.  I think that
> should be all.
>
> > - Correct me if I'm wrong, but one that comes to mind is when a server
> > reboots (after cleanly stopping a raid1 array that had a faulty
> > member) and then either:
> > 1) assembles the array with the previously faulty member now
> > available
> >
> > 2) assembles the array with the same faulty member missing.  The user
> > later re-adds the faulty member
> >
> > AFAIK both scenarios would bring about a full resync.
>
> Only if the drive is not recognised as the original member.
> Can you test this out and report a sequence of events that causes a
> full resync?

Sure, using an internal-bitmap-enabled raid1 with 2 loopback devices
on a stock 2.6.20.1 kernel, the following sequences result in a full
resync.  (FYI, I'm fairly certain I've seen this same behavior on
2.6.18 and 2.6.15 kernels too but would need to retest):

1)
mdadm /dev/md0 --manage --fail /dev/loop0
mdadm -S /dev/md0
mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1
  mdadm: /dev/md0 has been started with 1 drive (out of 2).
  NOTE: kernel log says:  md: kicking non-fresh loop0 from array!
mdadm /dev/md0 --manage --re-add /dev/loop0

2)
mdadm /dev/md0 --manage --fail /dev/loop0
mdadm /dev/md0 --manage --remove /dev/loop0
mdadm -S /dev/md0
mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1
  mdadm: /dev/md0 has been started with 1 drive (out of 2).
  NOTE: kernel log says:  md: kicking non-fresh loop0 from array!
mdadm /dev/md0 --manage --re-add /dev/loop0

Is stopping the MD (either with mdadm -S or a server reboot) tainting
that faulty member's ability to come back in using a quick
bitmap-based resync?

Mike

Re: Need clarification on raid1 resync behavior with bitmap support

[Neil Brown]

On Monday July 23, snitzer@gmail.com wrote:
> On 7/23/07, Neil Brown <neilb@suse.de> wrote:
> > Can you test this out and report a sequence of events that causes a
> > full resync?
> 
> Sure, using an internal-bitmap-enabled raid1 with 2 loopback devices
> on a stock 2.6.20.1 kernel, the following sequences result in a full
> resync.  (FYI, I'm fairly certain I've seen this same behavior on
> 2.6.18 and 2.6.15 kernels too but would need to retest):
> 
> 1)
> mdadm /dev/md0 --manage --fail /dev/loop0
> mdadm -S /dev/md0
> mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1
>   mdadm: /dev/md0 has been started with 1 drive (out of 2).
>   NOTE: kernel log says:  md: kicking non-fresh loop0 from array!
> mdadm /dev/md0 --manage --re-add /dev/loop0


sorry for the slow response.

It looks like commit 1757128438d41670ded8bc3bc735325cc07dc8f9
(December 2006) set conf->fullsync a litte too often.

This seems to fix it, and I'm fairly sure it is correct.

Thanks,
NeilBrown

----------------------------------
Make sure a re-add after a restart honours bitmap when resyncing.

Commit 1757128438d41670ded8bc3bc735325cc07dc8f9 was slightly bad.
If and array has a write-intent bitmap, and you remove a drive,
then readd it, only the changes parts should be resynced.
This only works if the array has not been shut down and restarted.

The above mentioned commit sets 'fullsync' at little more often
than it should.  This patch is more careful.

Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./drivers/md/raid1.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff .prev/drivers/md/raid1.c ./drivers/md/raid1.c
--- .prev/drivers/md/raid1.c	2007-07-31 11:21:29.000000000 +1000
+++ ./drivers/md/raid1.c	2007-08-03 16:24:23.000000000 +1000
@@ -1918,7 +1918,8 @@ static int run(mddev_t *mddev)
 		    !test_bit(In_sync, &disk->rdev->flags)) {
 			disk->head_position = 0;
 			mddev->degraded++;
-			conf->fullsync = 1;
+			if (disk->rdev)
+				conf->fullsync = 1;
 		}
 	}
 	if (mddev->degraded == conf->raid_disks) {

Re: Need clarification on raid1 resync behavior with bitmap support

[Mike Snitzer]

On 8/3/07, Neil Brown <neilb@suse.de> wrote:
> On Monday July 23, snitzer@gmail.com wrote:
> > On 7/23/07, Neil Brown <neilb@suse.de> wrote:
> > > Can you test this out and report a sequence of events that causes a
> > > full resync?
> >
> > Sure, using an internal-bitmap-enabled raid1 with 2 loopback devices
> > on a stock 2.6.20.1 kernel, the following sequences result in a full
> > resync.  (FYI, I'm fairly certain I've seen this same behavior on
> > 2.6.18 and 2.6.15 kernels too but would need to retest):
> >
> > 1)
> > mdadm /dev/md0 --manage --fail /dev/loop0
> > mdadm -S /dev/md0
> > mdadm --assemble /dev/md0 /dev/loop0 /dev/loop1
> >   mdadm: /dev/md0 has been started with 1 drive (out of 2).
> >   NOTE: kernel log says:  md: kicking non-fresh loop0 from array!
> > mdadm /dev/md0 --manage --re-add /dev/loop0
>
>
> sorry for the slow response.
>
> It looks like commit 1757128438d41670ded8bc3bc735325cc07dc8f9
> (December 2006) set conf->fullsync a litte too often.
>
> This seems to fix it, and I'm fairly sure it is correct.
>
> Thanks,
> NeilBrown
>
> ----------------------------------
> Make sure a re-add after a restart honours bitmap when resyncing.
>
> Commit 1757128438d41670ded8bc3bc735325cc07dc8f9 was slightly bad.
> If and array has a write-intent bitmap, and you remove a drive,
> then readd it, only the changes parts should be resynced.
> This only works if the array has not been shut down and restarted.
>
> The above mentioned commit sets 'fullsync' at little more often
> than it should.  This patch is more careful.

I hand-patched your change into a 2.6.20.1 kernel (I'd imagine your
patch is against current git).  I didn't see any difference because
unfortunately both of my full resync scenarios included stopping a
degraded raid after either: 1) having failed but not been removed a
member 2) having failed and removed a member.  In both scenarios if I
didn't stop the array and I just removed and re-added the faulty drive
the array would _not_ do a full resync.

My examples clearly conflict with your assertion that: "This only
works if the array has not been shut down and restarted."

But shouldn't raid1 be better about leveraging the bitmap of known
good (fresh) members even after having stopped a degraded array?  Why
is it that when an array is stopped raid1 seemingly loses the required
metadata that enables bitmap resyncs to just work upon re-add IFF the
array is _not_ stopped?  Couldn't raid1 be made to assemble the array
to look like the array had never been stopped, leaving the non-fresh
members out as it already does, and only then re-add the "non-fresh"
members that were provided?

To be explicit: isn't the bitmap still valid on the fresh members?  If
so, why is raid1 just disregarding the fresh bitmap?

Thanks, I really appreciate your insight.
Mike

Re: Need clarification on raid1 resync behavior with bitmap support

[Neil Brown]

On Friday August 3, snitzer@gmail.com wrote:
> 
> I hand-patched your change into a 2.6.20.1 kernel (I'd imagine your
> patch is against current git).  I didn't see any difference because
> unfortunately both of my full resync scenarios included stopping a
> degraded raid after either: 1) having failed but not been removed a
> member 2) having failed and removed a member.  In both scenarios if I
> didn't stop the array and I just removed and re-added the faulty drive
> the array would _not_ do a full resync.
> 
> My examples clearly conflict with your assertion that: "This only
> works if the array has not been shut down and restarted."

I think my changelog entry for the patch was poorly written.
What I meant to say was:
  *before this patch*  a remove and re-add only does a partial resync
    if the array has not been shutdown and restarted in the interim.
  The implication being that *after the patch*, a shutdown and restart
  will not interfere and a remove followed by a readd will always do a
  partial resync, even if the array was shutdown and restarted while
  degraded.

> 
> To be explicit: isn't the bitmap still valid on the fresh members?  If
> so, why is raid1 just disregarding the fresh bitmap?

Yes.  Exactly.  It is my understanding and experience that the patch I
sent fixes a bug so that it doesn't disregard the fresh bitmap.  It
should fix it for 2.6.20.1 as well.

Are you saying that you tried the same scenario with the patch applied
and it still did a full resync?  How do you measure whether it did a
full resync or a partial resync?

Thanks,
NeilBrown

Re: Need clarification on raid1 resync behavior with bitmap support

[Mike Snitzer]

On 8/3/07, Neil Brown <neilb@suse.de> wrote:
> On Friday August 3, snitzer@gmail.com wrote:
> >
> > I hand-patched your change into a 2.6.20.1 kernel (I'd imagine your
> > patch is against current git).  I didn't see any difference because
> > unfortunately both of my full resync scenarios included stopping a
> > degraded raid after either: 1) having failed but not been removed a
> > member 2) having failed and removed a member.  In both scenarios if I
> > didn't stop the array and I just removed and re-added the faulty drive
> > the array would _not_ do a full resync.
> >
> > My examples clearly conflict with your assertion that: "This only
> > works if the array has not been shut down and restarted."
>
> I think my changelog entry for the patch was poorly written.
> What I meant to say was:
>   *before this patch*  a remove and re-add only does a partial resync
>     if the array has not been shutdown and restarted in the interim.
>   The implication being that *after the patch*, a shutdown and restart
>   will not interfere and a remove followed by a readd will always do a
>   partial resync, even if the array was shutdown and restarted while
>   degraded.

Great, thanks for clarifying.

> > To be explicit: isn't the bitmap still valid on the fresh members?  If
> > so, why is raid1 just disregarding the fresh bitmap?
>
> Yes.  Exactly.  It is my understanding and experience that the patch I
> sent fixes a bug so that it doesn't disregard the fresh bitmap.  It
> should fix it for 2.6.20.1 as well.
>
> Are you saying that you tried the same scenario with the patch applied
> and it still did a full resync?  How do you measure whether it did a
> full resync or a partial resync?

I must not have loaded the patched raid1.ko because after retesting it
is clear that your patch does in fact fix the issue.  FYI, before, I
could just tell a full resync was occurring by looking at /proc/mdstat
and the time that elapsed.

Thanks for your help, any idea when this fix will make it upstream?

regards,
Mike

Re: Need clarification on raid1 resync behavior with bitmap support

[Neil Brown]

On Monday August 6, snitzer@gmail.com wrote:
> 
> Thanks for your help, any idea when this fix will make it upstream?
> 

Thanks for the confirmation.... yes, I guess I should send it upstream
soon.  Probably early next week at the latest.

NeilBrown

2 failed disks RAID 5 behavior bug?

[TJ Harrell]

Hi!

Let me apologize in advance for not having as much information as I'd like
to.

I have a RAID 5 array with 3 elements. Kernel is 2.6.23.

I had a SATA disk fail. On analysis, it's SMART claimed it had an
'electrical failure'. The drive sounded like an angry buzz-saw, so I'm
guessing more was going on with it. Anyway, when the drive failed,
/proc/mdstat showed two drives marked as failed [__U]. The other failed
drive was on the other channel of the same SATA controller. On inspection,
this second drive works fine. I'm guessing somehow the failing drive caused
the SATA controller to lock or something, which caused the RAID layer to
think the second drive was failed.

The problematic behavior is that once two elements were marked as failed,
any read or write access resulted in an "I/O Failure" message.
Unfortunately, I believe some writes were made to the array as the Event
Counter did not match on the two functional elements, and there was quite a
bit of data corruption of the superblock of the FS. 

I'm sorry I don't have more specifics, but I hope perhaps Mr. Brown or
someone else who knows the RAID code will consider making some sort of
safeguard to prevent writing to a RAID 5 array when more than one element is
failed.

PS: Please CC: me. :)

Thank You!
TJ Harrell
systemloc@verizon.net