[linux-lvm] LVM mirror resync

[linux-lvm] LVM mirror resync

[brem belguebli]

Hi,


I'm trying to find an explanation on how is handled drive failure and
how to recover a LVM mirror with on disk mirror log  from this kind of
failure.


Let me explain, I have this mirrored LV composed of:

LV_mimage_0 (on /dev/mpath/mpath0p2)
LV_mimage_1 (on /dev/mpath/mpath1p2)
and
LV_mlog (on /dev/mpath/mpath0p1)

When I simulate a drive failure (echo 1 > /sys/bus/scsi/devices/
"mpath1 Lunpaths"/delete ), the PV /dev/mpath/mpath1p1 is completely
removed from the VG.
The log is also removed from the VG, though it is located on the still
present physical drive.

When disk mpath1 is back again (echo "- - -"
/sys/class/scsi_host/HOST*/scan), it appears as not belonging to any
VG:

pvs giving the following output:
...
  /dev/mpath/mpath0p1 VG   lvm2 a-    36.00M  36.00M
  /dev/mpath/mpath0p2 VG   lvm2 a-     4.95G 256.00M
  /dev/mpath/mpath1p1        lvm2 --    39.19M  39.19M
  /dev/mpath/mpath1p2        lvm2 --     4.96G   4.96G
...

LV is monitored by dmeventd.

My questions are:

1) Am I using the right method to simulate a drive failure by echoing
a scsi delete ? How would behave the scsi layer in case a disk came to
physically disappear ?

2) If the above method is correct, isn't the role of the mirror log to
keep track of region changes  in case of a drive failure to be able to
incrementally resync the mirror when the missing drive is back to
operation?
 Missing drive is not replaced, the UUID is the same, the problem
could have occured at scsi connectivity not on the physical media (SAN
failure for instance).

3) if so, why does disappear the log device from the VG though being
on a physically present device ?

4) when the missing drive is back again,
    4-1)  why does it appear as not belonging to its original VG ?

    4-2) why doesn't it get automatically resync'ed by dmeventd ? .

The recover process seems to be to reconstruct from scratch the mirror
treating the failed and back again drive as a totally new drive.

Regards

Re: [linux-lvm] LVM mirror resync

[malahal]

brem belguebli [brem.belguebli@gmail.com] wrote:
> My questions are:
> 
> 1) Am I using the right method to simulate a drive failure by echoing
> a scsi delete ? How would behave the scsi layer in case a disk came to
> physically disappear ?

You are using a right method. You could pull the cables physically but
it is not going to make any difference to LVM mirror behaviour.

> 2) If the above method is correct, isn't the role of the mirror log to
> keep track of region changes  in case of a drive failure to be able to
> incrementally resync the mirror when the missing drive is back to
> operation?

Yes, but the implementation is not complete yet! Without the mirror
log (corelog), every reboot needs a complete resync. Mirror log is used
to avoid that but nothing more at this point!

>  Missing drive is not replaced, the UUID is the same, the problem
> could have occured at scsi connectivity not on the physical media (SAN
> failure for instance).
> 
> 3) if so, why does disappear the log device from the VG though being
> on a physically present device ?

If you really look at your LV, it is no more a mirror. It is just a
linear LV! So your failed mirror leg as well as your mirror log are not
used. "dmeventd" will remove the failed mirror leg from the VG. It
probably removed the now unused mirror log also.

> 4) when the missing drive is back again,
>     4-1)  why does it appear as not belonging to its original VG ?

Because it was removed from the VG in the first place. The metadata on
other disks is the latest and is used.

>     4-2) why doesn't it get automatically resync'ed by dmeventd ? .

It doesn't know any better! It doesn't know that it was a mirror.

> The recover process seems to be to reconstruct from scratch the mirror
> treating the failed and back again drive as a totally new drive.

Yes! I heard that mdadm (raid1) can do (optimal resyncs) what you want
but I have no direct experience. You may want to try that if you need
this feature.

Re: [linux-lvm] LVM mirror resync

[brem belguebli]

2009/9/24 �<malahal@us.ibm.com>:
> brem belguebli [brem.belguebli@gmail.com] wrote:
>> My questions are:
>>
>> 1) Am I using the right method to simulate a drive failure by echoing
>> a scsi delete ? How would behave the scsi layer in case a disk came to
>> physically disappear ?
>
> You are using a right method. You could pull the cables physically but
> it is not going to make any difference to LVM mirror behaviour.
>
>> 2) If the above method is correct, isn't the role of the mirror log to
>> keep track of region changes �in case of a drive failure to be able to
>> incrementally resync the mirror when the missing drive is back to
>> operation?
>
> Yes, but the implementation is not complete yet! Without the mirror
> log (corelog), every reboot needs a complete resync. Mirror log is used
> to avoid that but nothing more at this point!
>
>> �Missing drive is not replaced, the UUID is the same, the problem
>> could have occured at scsi connectivity not on the physical media (SAN
>> failure for instance).
>>
>> 3) if so, why does disappear the log device from the VG though being
>> on a physically present device ?
>
> If you really look at your LV, it is no more a mirror. It is just a
> linear LV! So your failed mirror leg as well as your mirror log are not
> used. "dmeventd" will remove the failed mirror leg from the VG. It
> probably removed the now unused mirror log also.
>
>> 4) when the missing drive is back again,
>> � � 4-1) �why does it appear as not belonging to its original VG ?
>
> Because it was removed from the VG in the first place. The metadata on
> other disks is the latest and is used.
>

The VG information is still present on the "failed and back" disk. This could
help the recovery process combined with next point to re assemble the mirror
and to eventually do partial resync.

>> � � 4-2) why doesn't it get automatically resync'ed by dmeventd ? .
>
> It doesn't know any better! It doesn't know that it was a mirror.

Keeping the log device in the VG wouldn't have helped it to keep track of this?


>> The recover process seems to be to reconstruct from scratch the mirror
>> treating the failed and back again drive as a totally new drive.
>
> Yes! I heard that mdadm (raid1) can do (optimal resyncs) what you want
> but I have no direct experience. You may want to try that if you need
> this feature.

Yes, I already use it. The setup is to configure an internal bitmap
(log area) on
your md array and it will do optimal resyncs in case of such a failure.
The thing is that I am doing this on a cluster and mdadm is not
naturally cluster aware.

Mirror log relocation doesn't seem to be implemented yet / working either  .

Regards

Re: [linux-lvm] LVM mirror resync

[brem belguebli]

Would it violate any principle to implement a third form of mirror
log, that would not require a 3rd device but still having the benefits
of keeping a persistent across reboots/crashes mirror status.

I'm thinking of a "internal" log (a la mdadm , HP-UX LVM, VXVM
mirroring, etc...)

Regards

2009/9/24 brem belguebli <brem.belguebli@gmail.com>:
> 2009/9/24 �<malahal@us.ibm.com>:
>> brem belguebli [brem.belguebli@gmail.com] wrote:
>>> My questions are:
>>>
>>> 1) Am I using the right method to simulate a drive failure by echoing
>>> a scsi delete ? How would behave the scsi layer in case a disk came to
>>> physically disappear ?
>>
>> You are using a right method. You could pull the cables physically but
>> it is not going to make any difference to LVM mirror behaviour.
>>
>>> 2) If the above method is correct, isn't the role of the mirror log to
>>> keep track of region changes �in case of a drive failure to be able to
>>> incrementally resync the mirror when the missing drive is back to
>>> operation?
>>
>> Yes, but the implementation is not complete yet! Without the mirror
>> log (corelog), every reboot needs a complete resync. Mirror log is used
>> to avoid that but nothing more at this point!
>>
>>> �Missing drive is not replaced, the UUID is the same, the problem
>>> could have occured at scsi connectivity not on the physical media (SAN
>>> failure for instance).
>>>
>>> 3) if so, why does disappear the log device from the VG though being
>>> on a physically present device ?
>>
>> If you really look at your LV, it is no more a mirror. It is just a
>> linear LV! So your failed mirror leg as well as your mirror log are not
>> used. "dmeventd" will remove the failed mirror leg from the VG. It
>> probably removed the now unused mirror log also.
>>
>>> 4) when the missing drive is back again,
>>> � � 4-1) �why does it appear as not belonging to its original VG ?
>>
>> Because it was removed from the VG in the first place. The metadata on
>> other disks is the latest and is used.
>>
>
> The VG information is still present on the "failed and back" disk. This could
> help the recovery process combined with next point to re assemble the mirror
> and to eventually do partial resync.
>
>>> � � 4-2) why doesn't it get automatically resync'ed by dmeventd ? .
>>
>> It doesn't know any better! It doesn't know that it was a mirror.
>
> Keeping the log device in the VG wouldn't have helped it to keep track of this?
>
>
>>> The recover process seems to be to reconstruct from scratch the mirror
>>> treating the failed and back again drive as a totally new drive.
>>
>> Yes! I heard that mdadm (raid1) can do (optimal resyncs) what you want
>> but I have no direct experience. You may want to try that if you need
>> this feature.
>
> Yes, I already use it. The setup is to configure an internal bitmap
> (log area) on
> your md array and it will do optimal resyncs in case of such a failure.
> The thing is that I am doing this on a cluster and mdadm is not
> naturally cluster aware.
>
> Mirror log relocation doesn't seem to be implemented yet / working either �.
>
> Regards
>