[linux-lvm] LVM mirror questions

[linux-lvm] LVM mirror questions

[Jay]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Hello,   

What happens to an lvm mirror if the disk log is lost ?   

I am having a hard time understanding Linux lvm ;  I created a mirror for my 
home system, two same-sized drives.    I decided to go for a mirror because I 
have too much space and I thought with the rash of failed hard drives I've had 
recently,  having a mirror would be ideal.     I had already installed my linux 
system, and I already had my /home filesystem in a single stripe/disk lv anyway, 
all I had to do was to add the other disk and lvcovert.    But it wouldn't work, 
saying I needed to add space on yet a third device for the log.   Either that, 
or I could keep the mirror log in memory.  Lvm would have to rebuild a 1TB 
mirror every time the system was rebooted if the log was left in memory.   That 
doesn't sound efficient.   So.   Put the log on disk.   I couldn't find an 
answer in man pages, but if the log is lost, then is the whole mirror lost ?   
That would mean to have the benefits of a mirror, the log would have to be at 
least as reliable as the mirror itself (or inconsequential).   

So, another mirror would have to be built of tiny size to hold the log, then 
lvconvert can be given something like -log disk /dev/vghuh/lvtiny, etc..

But then for the original mirror to be reliable, the second mirror would need a 
third mirror for the 2nd mirror's log.   Unless it is ok to leave the 2nd 
mirrors log in memory because it is so small.

What does everyone else do ?

Thank you in advance,
Jason Harris


      

[-- Attachment #2: Type: text/html, Size: 2021 bytes --]

Re: [linux-lvm] LVM mirror questions

[Petr Rockai]

Jay <jrharris19@yahoo.com> writes:
> What happens to an lvm mirror if the disk log is lost?

The mirror should continue to work with a core (memory) log.

Yours,
   Petr.

Re: [linux-lvm] LVM mirror questions

[Phillip Susi]

On 10/4/2010 7:55 PM, Jay wrote:
> the other disk and lvcovert.    But it wouldn't work, saying I needed to
> add space on yet a third device for the log.   Either that, or I could
> keep the mirror log in memory.  Lvm would have to rebuild a 1TB mirror
> every time the system was rebooted if the log was left in memory.   That
> doesn't sound efficient.   So.   Put the log on disk.   I couldn't find
> an answer in man pages, but if the log is lost, then is the whole mirror
> lost ?   That would mean to have the benefits of a mirror, the log would
> have to be at least as reliable as the mirror itself (or
> inconsequential).   
> 
> So, another mirror would have to be built of tiny size to hold the log,
> then lvconvert can be given something like -log disk /dev/vghuh/lvtiny,
> etc..

As you have noticed, LVM mirror support is basically broken or not fully
implemented yet.  The mirror log SHOULD be stored on both legs of the
mirror in a block allocated automatically in the metadata area by lvm,
but isn't.  If you want to mirror, you should do so using mdadm and then
use the md device as an lvm pv.

Re: [linux-lvm] LVM mirror questions

[Jay]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Phillip, 

Thank you for your answer and advice.  How unfortunate about lvm mirroring being 
broken.  I like your suggestion about how it SHOULD work.   In fact, long ago in 
a different age when I was working with rs6000s, I believe that is how reliable 
mirroring was accomplished with the ibm lvm.  Either that or it was that the 
metadata/log block was stored on multiple disks with a quorum needed to be 
functional.

I will convert to md mirror device.    If I had more skill, I would contribute 
somehow to the development effort.    As it is, I will lurk on the list to see I 
can be helpful to someone else.    Again, I appreciate your help.

Regards,
Jason Harris

 



________________________________
From: Phillip Susi <psusi@cfl.rr.com>
To: LVM general discussion and development <linux-lvm@redhat.com>
Cc: Jay <jrharris19@yahoo.com>
Sent: Tue, October 5, 2010 8:40:46 AM
Subject: Re: [linux-lvm] LVM mirror questions

On 10/4/2010 7:55 PM, Jay wrote:
> the other disk and lvcovert.    But it wouldn't work, saying I needed to
> add space on yet a third device for the log.   Either that, or I could
> keep the mirror log in memory.  Lvm would have to rebuild a 1TB mirror
> every time the system was rebooted if the log was left in memory.   That
> doesn't sound efficient.   So.   Put the log on disk.   I couldn't find
> an answer in man pages, but if the log is lost, then is the whole mirror
> lost ?   That would mean to have the benefits of a mirror, the log would
> have to be at least as reliable as the mirror itself (or
> inconsequential).  
> 
> So, another mirror would have to be built of tiny size to hold the log,
> then lvconvert can be given something like -log disk /dev/vghuh/lvtiny,
> etc..

As you have noticed, LVM mirror support is basically broken or not fully
implemented yet.  The mirror log SHOULD be stored on both legs of the
mirror in a block allocated automatically in the metadata area by lvm,
but isn't.  If you want to mirror, you should do so using mdadm and then
use the md device as an lvm pv.



      

[-- Attachment #2: Type: text/html, Size: 3062 bytes --]

Re: [linux-lvm] LVM mirror questions

[Jay]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

Hi Petr,

Thanks for the reply.  "Should continue to work.." leaves me with some doubt. 
 My intent for a mirror is for a datastore that has reliability at the expense 
of space.   In addition, when a large datastore is rebooted, there would be a 
period of time when the data is being mirrored, which leaves a window of 
peril. After I migrate my data off of lvm and onto an md device, I will test 
lvm. 

Test #1 will be start a huge write operation on a mirror that consists of 2 
mirrors and 1 log, all on seperate pvs.   While the write operation is 
happening, I will unplug the log drive and see if lvm switches in midstream to a 
memory log without loss.    

Test #2 will be to see if I can make lvm activate lvs or vgs in a certain order. 
  That way I could make the first 20 extents of every physical device part of a 
vg/lv that houses mirror logs.   These log lvs would be mirrored across all 
physical devices.   The log lvs would have memory logs, which would be ok since 
they will be so small.     The window of peril is reduced from hours to a few 
seconds. 

Anyways, thanks again for the answer!

Jason Harris



________________________________
From: Petr Rockai <prockai@redhat.com>
To: LVM general discussion and development <linux-lvm@redhat.com>
Sent: Tue, October 5, 2010 1:02:55 AM
Subject: Re: [linux-lvm] LVM mirror questions

Jay <jrharris19@yahoo.com> writes:
> What happens to an lvm mirror if the disk log is lost?

The mirror should continue to work with a core (memory) log.

Yours,
   Petr.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/



      

[-- Attachment #2: Type: text/html, Size: 2980 bytes --]

Re: [linux-lvm] LVM mirror questions

[Ray Morris]

   Please let us know the results of your mirror testing.
--
Ray Morris
support@bettercgi.com

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php


On 10/06/2010 10:50:41 AM, Jay wrote:
> Hi Petr,
> 
> Thanks for the reply.  "Should continue to work.." leaves me with  
> some doubt.
>  My intent for a mirror is for a datastore that has reliability at  
> the expense
> of space.   In addition, when a large datastore is rebooted, there  
> would be a
> period of time when the data is being mirrored, which leaves a window  
> of
> peril. After I migrate my data off of lvm and onto an md device, I  
> will test
> lvm.
> 
> Test #1 will be start a huge write operation on a mirror that  
> consists of 2
> mirrors and 1 log, all on seperate pvs.   While the write operation is
> happening, I will unplug the log drive and see if lvm switches in  
> midstream to a
> memory log without loss.
> 
> Test #2 will be to see if I can make lvm activate lvs or vgs in a  
> certain order.
>   That way I could make the first 20 extents of every physical device  
> part of a
> vg/lv that houses mirror logs.   These log lvs would be mirrored  
> across all
> physical devices.   The log lvs would have memory logs, which would  
> be ok since
> they will be so small.     The window of peril is reduced from hours  
> to a few
> seconds.
> 
> Anyways, thanks again for the answer!
> 
> Jason Harris
> 
> 
> 
> ________________________________
> From: Petr Rockai <prockai@redhat.com>
> To: LVM general discussion and development <linux-lvm@redhat.com>
> Sent: Tue, October 5, 2010 1:02:55 AM
> Subject: Re: [linux-lvm] LVM mirror questions
> 
> Jay <jrharris19@yahoo.com> writes:
> > What happens to an lvm mirror if the disk log is lost?
> 
> The mirror should continue to work with a core (memory) log.
> 
> Yours,
>    Petr.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 
> 
> 

------quoted attachment------
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM mirror questions

[Stuart D. Gathman]

On Wed, 6 Oct 2010, Jay wrote:

> Thank you for your answer and advice.  How unfortunate about lvm mirroring
> being broken.  I like your suggestion about how it SHOULD work.   In fact,
> long ago in a different age when I was working with rs6000s, I believe that
> is how reliable mirroring was accomplished with the ibm lvm.  Either that or

Yes, AIX mirroring was great.

The current LVM strategy could work, just use the option to store the 
mirror log on the same PV as one of the mirrors.  If you lose the PV
without the log - no problem.  If you lose the PV *with* the log, it
reverts to an in-core log, which is not a problem with the single remaining
leg.  The drawback to this is performance.  The main reason to put the
mirror log on a 3rd device is so the 3 writes occasioned by every write
to the LV can be done in parallel.

I don't know how LVM handles issues of PVs that are not permanently dead
and come back on line again later.  The md driver uses a generation count in
the super block to determine which is newer. (How is that updated without loss
of efficiency?)

I still use the md driver myself.  The LVM mirror support just doesn't "feel"
reliable yet.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] LVM mirror questions

[Petr Rockai]

Hi again,

Jay <jrharris19@yahoo.com> writes:

> Thanks for the reply.  "Should continue to work.." leaves me with some
> doubt.  My intent for a mirror is for a datastore that has reliability
> at the expense of space.  In addition, when a large datastore is
> rebooted, there would be a period of time when the data is being
> mirrored, which leaves a window of peril. After I migrate my data off
> of lvm and onto an md device, I will test lvm.

This depends on how you set up things. With the normal dmeventd
monitoring enabled, the mirror, when experiencing a log failure, will
discard the log and continue working with core log. Same for faulty
images (it'll discard the image, and depending on how many images you
have, continue as linear or with fewer images; with recent versions, you
can say in /etc/lvm/lvm.conf that you want the images/logs replaced
automatically from free PV space).

> Test #1 will be start a huge write operation on a mirror that consists
> of 2 mirrors and 1 log, all on seperate pvs.  While the write
> operation is happening, I will unplug the log drive and see if lvm
> switches in midstream to a memory log without loss.

If you have dmeventd running and the mirror volumes are monitored,
that's exactly what should happen. There might be a hitch in the IO flow
while the mirror is converted, but nothing should be lost. Please do let
us know if you observe anything else!

> Test #2 will be to see if I can make lvm activate lvs or vgs in a
> certain order.  That way I could make the first 20 extents of every
> physical device part of a vg/lv that houses mirror logs.  These log
> lvs would be mirrored across all physical devices.  The log lvs would
> have memory logs, which would be ok since they will be so small.  The
> window of peril is reduced from hours to a few seconds.

You can use --mirrorlog mirrored for that, you just need to play a bit
with the options to achieve the layout you want. Also please note that
having a log and a mirror image on the same drive may come with a
non-trivial performance penalty, since log updates now need to seek all
the way through the disk, on both images (regardless of how people think
things SHOULD work, the obvious is not always the best).

Yours,
   Petr.

PS: What happens when you don't have dmeventd running/monitoring enabled
probably depends on your kernel version. I believe it used to work in a
similar fashion without dmeventd (faulty log would be ignored), but I am
not sure that mode of operation is supported in newer versions. Someone
else might chime in knowing more about the kernel level details.

Re: [linux-lvm] LVM mirror questions

[Petr Rockai]

"Stuart D. Gathman" <stuart@bmsi.com> writes:
> The current LVM strategy could work, just use the option to store the 
> mirror log on the same PV as one of the mirrors.  If you lose the PV
> without the log - no problem.  If you lose the PV *with* the log, it
> reverts to an in-core log, which is not a problem with the single remaining
> leg.  The drawback to this is performance.  The main reason to put the
> mirror log on a 3rd device is so the 3 writes occasioned by every write
> to the LV can be done in parallel.

Yes, (write) performance is a concern. Otherwise, what you describe is
perfectly valid and achievable with LVM.

> I don't know how LVM handles issues of PVs that are not permanently dead
> and come back on line again later.  The md driver uses a generation count in
> the super block to determine which is newer. (How is that updated without loss
> of efficiency?)

LVM has a generation counter as well. It's only updated on metadata
writes though, so it doesn't cost anything. (Log is not part of metadata
in this sense.) When you lose a leg (and use dmeventd), the metadata on
the remaining PVs is updated to say that. The leg is also yanked from
the mirror. You can add it as a fresh image (with full resync) if it
ever comes back.

If you yank disks out of mirrors often (and plug them back afterwards),
you should maybe use replication instead of mirroring.

Yours,
   Petr.

Re: [linux-lvm] LVM mirror questions

[Phillip Susi]

On 10/6/2010 1:40 PM, Petr Rockai wrote:
> Yes, (write) performance is a concern. Otherwise, what you describe is
> perfectly valid and achievable with LVM.

It is?  What happens when the log device fails?

>> I don't know how LVM handles issues of PVs that are not permanently dead
>> and come back on line again later.  The md driver uses a generation count in
>> the super block to determine which is newer. (How is that updated without loss
>> of efficiency?)
> 
> LVM has a generation counter as well. It's only updated on metadata
> writes though, so it doesn't cost anything. (Log is not part of metadata
> in this sense.) When you lose a leg (and use dmeventd), the metadata on
> the remaining PVs is updated to say that. The leg is also yanked from
> the mirror. You can add it as a fresh image (with full resync) if it
> ever comes back.

Why a full resync?  With mdadm, it just keeps flagging the dirty chunks
so it only has to copy those when the other disk returns.

Re: [linux-lvm] LVM mirror questions

[Petr Rockai]

Phillip Susi <psusi@cfl.rr.com> writes:
>> LVM has a generation counter as well. It's only updated on metadata
>> writes though, so it doesn't cost anything. (Log is not part of metadata
>> in this sense.) When you lose a leg (and use dmeventd), the metadata on
>> the remaining PVs is updated to say that. The leg is also yanked from
>> the mirror. You can add it as a fresh image (with full resync) if it
>> ever comes back.
>
> Why a full resync?  With mdadm, it just keeps flagging the dirty chunks
> so it only has to copy those when the other disk returns.

Because it's really hard to guarantee anything when you experience a
failure. The safe route is to discard any data on the failed
drive. Certainly better than winding up with a subtly corrupt mirror.

As I said before, if you experience that a lot, you should be using
replication not mirroring.

Yours,
   Petr.