[linux-lvm] Simulating LVM Mirror Failure and recovery

[linux-lvm] Simulating LVM Mirror Failure and recovery

[yogeen honnavar]

Dear users,

We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.

now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards
-yogeen


      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Klaus Strebel]

yogeen honnavar schrieb:
> Dear users,
> 
> We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.
> 
> now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
> 
> dd if=/dev/zero of=/dev/vg1/lv1 count=10
> 
> as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
> 
> can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
> 
> thanks and regards
> -yogeen

Hi yogeen,

i guess you wanted to do a 'dd if=/dev/zero of=/dev/sdb3 count=10', with
your command, you filled the logical volume with zeros and that is
mirrored perfectly on both underlying devices - as it should ;-).



-- 
Mit freundlichen Grüssen / best regards

Klaus Strebel, Dipl.-Inform. (FH), mailto:klaus.strebel@gmx.net

/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[vu pham]

yogeen honnavar wrote:
> Dear users,
> 
> We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.
> 
> now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
> 
> dd if=/dev/zero of=/dev/vg1/lv1 count=10
> 
> as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
> 
> can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
> 
> thanks and regards

I think to simulate error failure you have to write to the underlying 
devices, not the mirrored device. When you write to the mirrored device, 
lvl1 in this case, data is mirrored so there is no failure at all. So I 
think you have to dd to either /dev/sda3 or /dev/sdb3.

Vu

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Tomasz Chmielewski]

vu pham schrieb:

>> as per the redhat guide this should result in mirror leg failure and 
>> any subsequent write activity to the mirror should convert the mirror 
>> into linear volume. but the output of lvs command still shows copy % 
>> as 100%. also data in filesystem of lv1 is erased and it is converted 
>> into read only filesystem.
>>
>> can anyone please let us know the correct way to simulate lvm mirror 
>> failure and recovery.
>>
>> thanks and regards
> 
> I think to simulate error failure you have to write to the underlying 
> devices, not the mirrored device. When you write to the mirrored device, 
> lvl1 in this case, data is mirrored so there is no failure at all. So I 
> think you have to dd to either /dev/sda3 or /dev/sdb3.

How will LVM distinguish between a valid and invalid copy in that case?

Are you sure that on reads, LVM will return data from the "good" copy, 
not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb 
seek=xxx bs=1M count=10"?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Alasdair G Kergon]

On Thu, Apr 16, 2009 at 07:07:57AM -0500, vu pham wrote:
> yogeen honnavar wrote:
> >Dear users,
> >dd if=/dev/zero of=/dev/vg1/lv1 count=10
> >as per the redhat guide this should result in mirror leg failure and any 

Can you provide the reference - where does it say this?

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Lupe Christoph]

On Thursday, 2009-04-16 at 15:47:07 +0530, yogeen honnavar wrote:

> dd if=/dev/zero of=/dev/vg1/lv1 count=10

> as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

Apart from the problem with the of= parameter that was already pointed
out - why do you assume writing *anything* *anywhere* would simulate a
mirror failure? The most you do simulate with that is mirror
*corruption*.

To simulate a failure, you have to somehow disable a drive. Unplug a
cable, force it to spin down (assuming that it is not automatically spun
up again, I dunno), change the zoning, whatever is needed to prevent
access to the device.

> can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

Depends a lot on your hardware. E.g. the system I use for testing are
about 400 km away, so I'd rather not unplug anything...

HTH,
Lupe Christoph
-- 
| There is no substitute for bad design except worse design.                   |
| /me                                                                          |

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Eric Brunson]

On 04/16/2009 06:54 AM, Alasdair G Kergon wrote:
> On Thu, Apr 16, 2009 at 07:07:57AM -0500, vu pham wrote:
>    
>> yogeen honnavar wrote:
>>      
>>> Dear users,
>>> dd if=/dev/zero of=/dev/vg1/lv1 count=10
>>> as per the redhat guide this should result in mirror leg failure and any
>>>        
>
> Can you provide the reference - where does it say this?
>
> Alasdair
>    

I believe this is the reference.

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Their example may be in error.

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Eric Brunson]

On 04/16/2009 07:36 AM, Eric Brunson wrote:
> On 04/16/2009 06:54 AM, Alasdair G Kergon wrote:
>> On Thu, Apr 16, 2009 at 07:07:57AM -0500, vu pham wrote:
>>> yogeen honnavar wrote:
>>>> Dear users,
>>>> dd if=/dev/zero of=/dev/vg1/lv1 count=10
>>>> as per the redhat guide this should result in mirror leg failure 
>>>> and any
>>
>> Can you provide the reference - where does it say this?
>>
>> Alasdair
>
> I believe this is the reference.
>
> http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html 
>
>
> Their example may be in error.
>
>

Upon review, I think it's correct, but worded awkwardly:

    In this example, the primary leg of the mirror |/dev/sda1| fails.
    Any write activity to the mirrored volume causes LVM to detect the
    failed mirror. When this occurs, LVM converts the mirror into a
    single linear volume. In this case, to trigger the conversion, we
    execute a |dd| command

    [root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|*
    10+0 records in
    10+0 records out


I believe the dd is not to cause the failure, it's simply to generate 
write activity to the volume group after the mirror leg has been caused 
to fail through other means not documented.

Anyone else think that's the case?

e.

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Don Bishop]

[-- Attachment #1: Type: text/plain, Size: 2631 bytes --]

Hey Eric

Nice to see your still around. How's it all going?

best,
Don B



On Thu, 2009-04-16 at 07:58 -0600, Eric Brunson wrote:

> On 04/16/2009 07:36 AM, Eric Brunson wrote:
> > On 04/16/2009 06:54 AM, Alasdair G Kergon wrote:
> >> On Thu, Apr 16, 2009 at 07:07:57AM -0500, vu pham wrote:
> >>> yogeen honnavar wrote:
> >>>> Dear users,
> >>>> dd if=/dev/zero of=/dev/vg1/lv1 count=10
> >>>> as per the redhat guide this should result in mirror leg failure 
> >>>> and any
> >>
> >> Can you provide the reference - where does it say this?
> >>
> >> Alasdair
> >
> > I believe this is the reference.
> >
> > http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html 
> >
> >
> > Their example may be in error.
> >
> >
> 
> Upon review, I think it's correct, but worded awkwardly:
> 
>     In this example, the primary leg of the mirror |/dev/sda1| fails.
>     Any write activity to the mirrored volume causes LVM to detect the
>     failed mirror. When this occurs, LVM converts the mirror into a
>     single linear volume. In this case, to trigger the conversion, we
>     execute a |dd| command
> 
>     [root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|*
>     10+0 records in
>     10+0 records out
> 
> 
> I believe the dd is not to cause the failure, it's simply to generate 
> write activity to the volume group after the mirror leg has been caused 
> to fail through other means not documented.
> 
> Anyone else think that's the case?
> 
> e.
> 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 

CONFIDENTIALITY NOTICE
This electronic mail transmission and any accompanying documents contain
information belonging to the sender that may be confidential and legally
privileged.  This information is intended only for the use of the
individual or entity to whom this electronic mail transmission was sent
as indicated above.  If you are not the intended recipient, any
disclosure, copying, distribution, or action taken in reliance on the
contents of the information contained in this transmissions is strictly
prohibited.  If you have received this transmission in error, please
reply by e-mail to let me know and then permanently delete the message.
Thank you.

[-- Attachment #2: Type: text/html, Size: 3694 bytes --]

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Lupe Christoph]

On Thursday, 2009-04-16 at 07:58:31 -0600, Eric Brunson wrote:

> Upon review, I think it's correct, but worded awkwardly:

Maybe it's because I'm not a native speaker, but I find it difficult to
understand this differently.

>    In this example, the primary leg of the mirror |/dev/sda1| fails.

It fails. Not "we make it fail". No indication what makes it fail.

>    Any write activity to the mirrored volume causes LVM to detect the
>    failed mirror. When this occurs, LVM converts the mirror into a
>    single linear volume. In this case, to trigger the conversion, we
>    execute a |dd| command

"we execute a |dd| command" "to trigger the *conversion*". Not to make
it fail.

> I believe the dd is not to cause the failure, it's simply to generate  
> write activity to the volume group after the mirror leg has been caused  
> to fail through other means not documented.

... as they said ...

So the dd command the OP sent was correct after all. He just
misunderstood its purpose.

Now back to our original topic - how do you make a mirror copy fail? ;-)

Lupe Christoph
-- 
| There is no substitute for bad design except worse design.                   |
| /me                                                                          |

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[vu pham]

Tomasz Chmielewski wrote:
> vu pham schrieb:
> 
>>> as per the redhat guide this should result in mirror leg failure and 
>>> any subsequent write activity to the mirror should convert the mirror 
>>> into linear volume. but the output of lvs command still shows copy % 
>>> as 100%. also data in filesystem of lv1 is erased and it is converted 
>>> into read only filesystem.
>>>
>>> can anyone please let us know the correct way to simulate lvm mirror 
>>> failure and recovery.
>>>
>>> thanks and regards
>>
>> I think to simulate error failure you have to write to the underlying 
>> devices, not the mirrored device. When you write to the mirrored 
>> device, lvl1 in this case, data is mirrored so there is no failure at 
>> all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.
> 
> How will LVM distinguish between a valid and invalid copy in that case?
> 
> Are you sure that on reads, LVM will return data from the "good" copy, 
> not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb 
> seek=xxx bs=1M count=10"?

Good question. I thought that dd will destroy the lvm information at the 
beginning of the partition which makes the leg broken. I made the 
following test and the result is odd:

- created mirrored logical volumn mrlv1 on /dev/sdc1 and /dev/sdd1, 
having /dev/sdd2 as log device
- mkfs.ext3 /dev/vg1/mrlv1, mount it and copy files onto it.
- dd if=/dev/zero of=/dev/sdd1
- read / write onto the mounted mirror lvm.

The output of lvs  shows:
- lvm information on /dev/sdd2 is destroyed
- the 2nd mirror, which is /dev/sdd2, is *unknown device*
- the mirror is still ok

And no lvm log in /var/log/messages.

[root@xen3 ~]# lvs -a -o +devices
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
   LV               VG   Attr   LSize   Origin Snap%  Move Log
[... deleted unrelated devices ...]

   mrlv1            vg1  mwi-ao 200.00M                    mrlv1_mlog 
100.00         mrlv1_mimage_0(0),mrlv1_mimage_1(0)
   [mrlv1_mimage_0] vg1  iwi-ao 200.00M 
              /dev/sdc1(0)
   [mrlv1_mimage_1] vg1  iwi-ao 200.00M 
              unknown device(0)
   [mrlv1_mlog]     vg1  lwi-ao   4.00M 
              /dev/sdd2(0)



Vu

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[malahal]

Lupe Christoph [lupe@lupe-christoph.de] wrote:
> On Thursday, 2009-04-16 at 15:47:07 +0530, yogeen honnavar wrote:
>
> > dd if=/dev/zero of=/dev/vg1/lv1 count=10
>
> > as per the redhat guide this should result in mirror leg failure and any
subsequent write activity to the mirror should convert the mirror into linear
volume. but the output of lvs command still shows copy % as 100%. also data in
filesystem of lv1 is erased and it is converted into read only filesystem.
>
> Apart from the problem with the of= parameter that was already pointed
> out - why do you assume writing *anything* *anywhere* would simulate a
> mirror failure? The most you do simulate with that is mirror
> *corruption*.

You are absolutely right.

>> can anyone please let us know the correct way to simulate lvm mirror failure
>> and recovery.
>
> Depends a lot on your hardware. E.g. the system I use for testing are
> about 400 km away, so I'd rather not unplug anything...

I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda
disappear and fail for LVM tests. You can fail the disk in a lot of ways
depending on your hardware, but this trick works with almost every
device!

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[Takahiro Yasui]

malahal@us.ibm.com wrote:
> I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda
> disappear and fail for LVM tests. You can fail the disk in a lot of ways
> depending on your hardware, but this trick works with almost every
> device!

# echo offline > /sys/block/sda/device/state

might work as well. In this case, you can recover the failure by

# echo running > /sys/block/sda/device/state

Thanks,
Taka

Re: [linux-lvm] Simulating LVM Mirror Failure and recovery

[yogeen honnavar]

hi all,

Thanks for all the replies. as pointed out by Eric brunson i was referring to the following document

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html 

i used the below command to simulate failed sda and then wrote some data to the mounted file system to trigger LVM conversion to linear mode.

# echo offline > /sys/block/sda/device/state 

thanks to all.
regards
-yogeen

--- On Thu, 16/4/09, Takahiro Yasui <tyasui@redhat.com> wrote:

> From: Takahiro Yasui <tyasui@redhat.com>
> Subject: Re: [linux-lvm] Simulating LVM Mirror Failure and recovery
> To: "LVM general discussion and development" <linux-lvm@redhat.com>
> Date: Thursday, 16 April, 2009, 9:32 PM
> malahal@us.ibm.com
> wrote:
> > I usually run "echo 1 >
> /sys/block/sda/device/delete" to make /dev/sda
> > disappear and fail for LVM tests. You can fail the
> disk in a lot of ways
> > depending on your hardware, but this trick works with
> almost every
> > device!
> 
> # echo offline > /sys/block/sda/device/state
> 
> might work as well. In this case, you can recover the
> failure by
> 
> # echo running > /sys/block/sda/device/state
> 
> Thanks,
> Taka
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 


      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/