[linux-lvm] Mirror resync direction

[linux-lvm] Mirror resync direction

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

I'm running kernel 2.6.17 using the lvm2 package currently in debian
testing (2.02.06-3). I use it to set up a VG containing two
PVs, /dev/sda2 and /dev/sdb2. I create a mirror LV using corelog:

  lvcreate --mirror 1 --corelog ...

I create a filesystem on it, and populate it with files. I then try
testing how it copes with drive failures, by simulating a replaced disk.

To do this, I zero out one of the sides using 

  dd if=/dev/zero of=/dev/sda2

Having done that, I do the usual restore method

  pvcreate --uuid 'whatever' /dev/sda2
  vgcfgrestore vg
  vgchange -a y

And I note that it starts to resync the mirror. But it goes in the
"wrong" direction - sda2's PV is the master, but that's been zeroed out.
sdb2's is the slave but contains the real data. The block copying goes in
the wrong direction here, copying the master's zeroes over the good data
in the slave.

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it
perhaps a limitation of --corelog; in which case, how can I protect
against that?


-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Mirror resync direction

[Alasdair G Kergon]

On Thu, Oct 26, 2006 at 09:17:38PM +0100, Paul LeoNerd Evans wrote:
> testing (2.02.06-3). I use it to set up a VG containing two

> Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it
> perhaps a limitation of --corelog; in which case, how can I protect
> against that?

Firstly, this is work in progress, so you should be using newer
tools.  Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the
lost device, add the replacement disk to the VG in the normal
way, then use lvconvert to turn the LV back into a mirror.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Mirror resync direction

[Alasdair G Kergon]

On Thu, Oct 26, 2006 at 09:27:02PM +0100, Alasdair G Kergon wrote:
> Firstly, this is work in progress, so you should be using newer
> tools.  Secondly, with newer tools you should follow a different
> procedure: use vgreduce --removemissing (carefully) to remove the
> lost device, add the replacement disk to the VG in the normal
> way, then use lvconvert to turn the LV back into a mirror.
 
And I should add, dmeventd will do these steps automatically on failure if you
configure it into the lvm2 and device-mapper packages and apply the right
kernel patches (they're nearly ready for upstream submission).

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Mirror resync direction

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Thu, 26 Oct 2006 21:27:02 +0100
Alasdair G Kergon <agk@redhat.com> wrote:

> On Thu, Oct 26, 2006 at 09:17:38PM +0100, Paul LeoNerd Evans wrote:
> > testing (2.02.06-3). I use it to set up a VG containing two
> 
> > Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it
> > perhaps a limitation of --corelog; in which case, how can I protect
> > against that?
> 
> Firstly, this is work in progress, so you should be using newer
> tools.  Secondly, with newer tools you should follow a different
> procedure: use vgreduce --removemissing (carefully) to remove the
> lost device, add the replacement disk to the VG in the normal
> way, then use lvconvert to turn the LV back into a mirror.

Thanks for the quick reply :)

I've looked at my copy of vgreduce, and that has the option you named. I
tried that and it did as I suspected, namely converted the mirror back to
a plain linear LV, that I can mount as normal and use. Then running
pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.

Thanks for the help.

> And I should add, dmeventd will do these steps automatically on failure
> if you configure it into the lvm2 and device-mapper packages and apply
> the right kernel patches (they're nearly ready for upstream submission).

Will that be a separate userland program? Presumably it'll appear in a
debian package of the same name at some point.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Mirror resync direction

[Jonathan E Brassow]

On Oct 26, 2006, at 4:10 PM, Paul LeoNerd Evans wrote:

>
>> And I should add, dmeventd will do these steps automatically on 
>> failure
>> if you configure it into the lvm2 and device-mapper packages and apply
>> the right kernel patches (they're nearly ready for upstream 
>> submission).
>
> Will that be a separate userland program? Presumably it'll appear in a
> debian package of the same name at some point.
>

dmeventd will be a separate userland binary, but will be part of the 
device-mapper package.

  brassow

Re: [linux-lvm] Mirror resync direction

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 1686 bytes --]

On Thu, 26 Oct 2006 21:27:02 +0100
Alasdair G Kergon <agk@redhat.com> wrote:

> On Thu, Oct 26, 2006 at 09:17:38PM +0100, Paul LeoNerd Evans wrote:
> > testing (2.02.06-3). I use it to set up a VG containing two
> 
> > Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it
> > perhaps a limitation of --corelog; in which case, how can I protect
> > against that?
> 
> Firstly, this is work in progress, so you should be using newer
> tools.  Secondly, with newer tools you should follow a different
> procedure: use vgreduce --removemissing (carefully) to remove the
> lost device, add the replacement disk to the VG in the normal
> way, then use lvconvert to turn the LV back into a mirror.

Thanks for the quick reply :)

I've looked at my copy of vgreduce, and that has the option you named. I
tried that and it did as I suspected, namely converted the mirror back to
a plain linear LV, that I can mount as normal and use. Then running
pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.

Thanks for the help.

> And I should add, dmeventd will do these steps automatically on failure
> if you configure it into the lvm2 and device-mapper packages and apply
> the right kernel patches (they're nearly ready for upstream submission).

Will that be a separate userland program? Presumably it'll appear in a
debian package of the same name at some point.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/


-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Mirror resync direction

[Jonathan E Brassow]

On Oct 26, 2006, at 3:17 PM, Paul LeoNerd Evans wrote:

> I'm running kernel 2.6.17 using the lvm2 package currently in debian
> testing (2.02.06-3). I use it to set up a VG containing two
> PVs, /dev/sda2 and /dev/sdb2. I create a mirror LV using corelog:
>
>   lvcreate --mirror 1 --corelog ...
>
> I create a filesystem on it, and populate it with files. I then try
> testing how it copes with drive failures, by simulating a replaced 
> disk.
>
> To do this, I zero out one of the sides using
>
>   dd if=/dev/zero of=/dev/sda2

This doesn't really reflect what would happen if you replaced a disk, I 
don't think.  With this, you are simply going around mirror's back to 
write data to a device which it controls.

In real life, I think you would have a disk failure (which would force 
you to run 'vgreduce --removemissing vg'), then you would insert a new 
block device, pvcreate/vgextend/lvconvert.

(When dmeventd is properly running, it should take care of the steps 
that are currently manual.)

> Having done that, I do the usual restore method
>
>   pvcreate --uuid 'whatever' /dev/sda2
>   vgcfgrestore vg
>   vgchange -a y
>
> And I note that it starts to resync the mirror. But it goes in the
> "wrong" direction - sda2's PV is the master, but that's been zeroed 
> out.
> sdb2's is the slave but contains the real data. The block copying goes 
> in
> the wrong direction here, copying the master's zeroes over the good 
> data
> in the slave.

Again, mirror has no idea that you did this.  The act of dd'ing to an 
underlying device is like dd'ing to a device under a file system - it 
doesn't know you are doing it and causes corruption.

> Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is 
> it
> perhaps a limitation of --corelog; in which case, how can I protect
> against that?
>

LVM2 mirroring is still a work in progress (trying to get the right 
pieces upstream).  Depending on the version of LVM2 and the kernel 
patches you have, it may not work.

The use of --corelog simply means that the mirror device will be 
resynchronized every time the device is activated.  The persistent log 
(disk log) tracks what resynchronization has been done and can avoid 
all complete resyncs.

  brassow

Re: [linux-lvm] Mirror resync direction

[Paul LeoNerd Evans]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Thu, 26 Oct 2006 16:14:34 -0500
Jonathan E Brassow <jbrassow@redhat.com> wrote:

> This doesn't really reflect what would happen if you replaced a disk, I 
> don't think.  With this, you are simply going around mirror's back to 
> write data to a device which it controls.

Well, I did it with the VG turned off; namely

  vgchange -a n

> In real life, I think you would have a disk failure (which would force 
> you to run 'vgreduce --removemissing vg'), then you would insert a new 
> block device, pvcreate/vgextend/lvconvert.

Yes, I wasn't aware of that procedure before, but Alasdair Kergon's mail
has pointed that out to me - I've tried with that and it seems to work
fine.

> LVM2 mirroring is still a work in progress (trying to get the right 
> pieces upstream).  Depending on the version of LVM2 and the kernel 
> patches you have, it may not work.

Seems happy once I follow the correct procedure - see above :)

> The use of --corelog simply means that the mirror device will be 
> resynchronized every time the device is activated.  The persistent log 
> (disk log) tracks what resynchronization has been done and can avoid 
> all complete resyncs.

Yes, I'm aware of that bit, but I couldn't see any other way to achieve
it with only two PVs. The "--alloc anywhere" as suggested in the error
message doesn't have any effect for me.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Mirror resync direction

[Jonathan E Brassow]

On Oct 26, 2006, at 4:16 PM, Paul LeoNerd Evans wrote:
>> The use of --corelog simply means that the mirror device will be
>> resynchronized every time the device is activated.  The persistent log
>> (disk log) tracks what resynchronization has been done and can avoid
>> all complete resyncs.
>
> Yes, I'm aware of that bit, but I couldn't see any other way to achieve
> it with only two PVs. The "--alloc anywhere" as suggested in the error
> message doesn't have any effect for me.
>

Jun'ichi Nomura from NEC is working on the allocation issues - it's a 
bit buggy WRT mirrors.

However, you could use a known bug to get around having to use 
--corelog when you only have two devices.

1) lvcreate -m1 --corelog -L <don't use all the space> -n mirror vg
2) lvconvert -m1 vg/mirror

The convert doesn't currently honor the allocation schemes very well 
right now either.  It will simply put the disk log on one of the two 
devices.

  brassow