[linux-lvm] Fun and games with mirroring

[linux-lvm] Fun and games with mirroring

[Vic]

Hi All.

Forgive me if this has been dealt with before - I couldn't find a search
tool for the archives...

I've just had a play with LVM mirroring. It appears to have a rather
serious usability issue.

I added a new drive to my system, created a PV, then used lvconvert -m1 to
mirror my various volumes. Sure enough, the new PV eventually contained my
data. As far as I knew, I now had failure resilience at the LVM layer.

So I simulated a failure - I powered down the machine and removed the new
drive. And then it all went wrong.

The boot sequence went through Grub, then failed. It could not find my
root FS. Swapping drives to the other one did exactly the same - neither
disk from my "mirror" was bootable.

After a bit of poking at it, I found that the LVM system would not start
the VG with a missing PV. Even after starting it manually (and mounting
it, and looking at it) at a recovery shell (dracut? I'm not sure), I still
couldn't get the machine booted.

I fixed it eventually by booting into a LiveUSB. The only thing I could
get to work was to vgreduce --removemissing --force each of the disks.
This is clearly not the sort of thing we ought to be doing for something
as simple as a disk failure...

So it is that using LVM mirrors, we end up with the machine being rendered
unbootable if either disk fails. That can't be right, can it?

Vic.

p.s. The reason I want to do this with LVM mirrors rather than MD RAID is
that this is potentially a very useful way of grabbing a backup snapshot
of a machine prior to some risky operation - it allows me to "fork" a
machine onto two sets of media. But if it leaves the result unbootable,
that's something of a problem...

Re: [linux-lvm] Fun and games with mirroring

[Les Mikesell]

On Tue, May 22, 2012 at 9:50 AM, Vic <lvm@beer.org.uk> wrote:
>
>
> p.s. The reason I want to do this with LVM mirrors rather than MD RAID is
> that this is potentially a very useful way of grabbing a backup snapshot
> of a machine prior to some risky operation - it allows me to "fork" a
> machine onto two sets of media. But if it leaves the result unbootable,
> that's something of a problem...
>

How is it different/better than 'mdadm --fail' ing a member out of an
MD raid1 set as a temporary snapshot?

-- 
  Les Mikesell
    lesmikesell@gmail.com

Re: [linux-lvm] Fun and games with mirroring

[Vic]

> How is it different/better than 'mdadm --fail' ing a member out of an
> MD raid1 set as a temporary snapshot?

You can do it to a box that hasn't been set up with mdadm...

Vic.

Re: [linux-lvm] Fun and games with mirroring

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/22/2012 04:29 PM, Vic wrote:
> 
>> How is it different/better than 'mdadm --fail' ing a member out
>> of an MD raid1 set as a temporary snapshot?
> 
> You can do it to a box that hasn't been set up with mdadm...

Sure but you will need LVM set up. In that case, why not use LVM's
snapshot capabilities? If you're using a distribution with a recent
kernel then you'll have access to the new shared snapshot
implementation that gives far better performance than the old
non-shared one.

Some distributions (yum based) even include a package updater plug-in
that will automatically take snapshots before applying a set of
package updates and allow you to either rollback or merge the change
after testing.

You might find this works better for this particular use of mirroring
but you can also try the --splitmirrors option to the lvconvert
command (but it will be less efficient since it will take a
block-by-block copy of the origin during the lvconvert -m1 rather than
just recording changes as a snapshot would). Mirror splitting was
added in 2.02.88 (about nine months old).

I've found that dracut doesn't always handle LVM errors particularly
gracefully (in particular if there is a problem the dracut main loop
appears to reset state after leaving the interactive shell so even if
you do fix the problem such that the system could continue to boot it
will just fail in the same way again) and by the sound of it this
could be improved for your case too.

Ideally you'd want the initramfs scripts to carry out the same actions
as the defined mirror repair policy during startup (or to invoke
lvconvert --repair to do the same).

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+7uDQACgkQ6YSQoMYUY95ECwCgkcYwE6ujXi5rLOApVZg5wSWN
pj8An05qCXCcaOSt+sgOZvpYORKHU4gM
=zW6z
-----END PGP SIGNATURE-----

Re: [linux-lvm] Fun and games with mirroring

[Vic]

>> You can do it to a box that hasn't been set up with mdadm...
>
> Sure but you will need LVM set up.

I always have.

> In that case, why not use LVM's snapshot capabilities?

Because that doesn't do what I want.

I'm not looking to have a LVM snapshot. I'm looking to have a duplicate
disk. This allows me to experiment on one, whilst keeping the other safe.
Alternatively, it allows me to migrate from one disk to another without
too much risk - before anyone points me at pvmove, I lost a filesystem
when that fell over on a live machine once :-(

> You might find this works better for this particular use of mirroring
> but you can also try the --splitmirrors option to the lvconvert
> command (but it will be less efficient since it will take a
> block-by-block copy of the origin during the lvconvert -m1 rather than
> just recording changes as a snapshot would). Mirror splitting was
> added in 2.02.88 (about nine months old).

--splitmirrors would seem to do what I'm after *except* that it will leave
the mirror unbootable without further attention (the machine won't boot
with the wrong name on the LVs)

> I've found that dracut doesn't always handle LVM errors particularly
> gracefully (in particular if there is a problem the dracut main loop
> appears to reset state after leaving the interactive shell so even if
> you do fix the problem such that the system could continue to boot it
> will just fail in the same way again) and by the sound of it this
> could be improved for your case too.

This is true - but sorta misses the point; a LV mirror is not a LV mirror
if it won't activate. Adding a mirror means that a faliure of *either*
side renders the whole unusable.

> Ideally you'd want the initramfs scripts to carry out the same actions
> as the defined mirror repair policy during startup (or to invoke
> lvconvert --repair to do the same).

Well, we can call it Somebody Else's Problem, or perhaps LVM should be
able to use a broken mirror - that's how every other mirror system works,
and is generally the very reason for having a mirror at all. I consider
this to be a bug within LVM.

Vic.

Re: [linux-lvm] Fun and games with mirroring

[Brassow Jonathan]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]


On May 22, 2012, at 11:40 AM, Vic wrote:

>> 
>> Ideally you'd want the initramfs scripts to carry out the same actions
>> as the defined mirror repair policy during startup (or to invoke
>> lvconvert --repair to do the same).
> 
> Well, we can call it Somebody Else's Problem, or perhaps LVM should be
> able to use a broken mirror - that's how every other mirror system works,
> and is generally the very reason for having a mirror at all. I consider
> this to be a bug within LVM.

LVM requires that for an LV to be activated while there are missing PVs, the '--partial' flag must be used.  If the script running vgchange does not include '--partial', your mirror won't be activated.

There are also 2 different ways of doing mirroring in LVM now also.  You can choose the 'mirror' segment type or the 'raid1' segment type.  The 'raid1' segment type is a newer offering, but leverages the MD RAID1 personality.  The critical difference between the two is that 'raid1' can operate while there is a faulty device present and the 'mirror' segment type must have something intervene and remove/replace the failed device before it will allow I/O.

It seems to me that the solution might be to attempt to activate the volume group without the partial flag first.  Failing that, activate any 'raid1' LV with a --partial flag and repair and activate any 'mirror' LVs.

 brassow

[-- Attachment #2: Type: text/html, Size: 2411 bytes --]

Re: [linux-lvm] Fun and games with mirroring

[Vic]

> LVM requires that for an LV to be activated while there are missing PVs,
> the '--partial' flag must be used.  If the script running vgchange does
> not include '--partial', your mirror won't be activated.

I tried that. It didn't work.

Vic.

Re: [linux-lvm] Fun and games with mirroring

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/22/2012 08:59 PM, Vic wrote:
> 
>> LVM requires that for an LV to be activated while there are
>> missing PVs, the '--partial' flag must be used.  If the script
>> running vgchange does not include '--partial', your mirror won't
>> be activated.
> 
> I tried that. It didn't work.

Maybe try reporting what happened and where it went wrong so that
others can try to help you? Just stating that you consider something
a bug doesn't really help to move things forward.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+8pVUACgkQ6YSQoMYUY96AkgCfWjDsChbxXaC46+XMxFW8EFvL
0/0An1K00il83NSow4qS31lHsROq64LA
=VuGB
-----END PGP SIGNATURE-----

Re: [linux-lvm] Fun and games with mirroring

[Zdenek Kabelac]

Dne 22.5.2012 21:59, Vic napsal(a):
>
>> LVM requires that for an LV to be activated while there are missing PVs,
>> the '--partial' flag must be used.  If the script running vgchange does
>> not include '--partial', your mirror won't be activated.
>
> I tried that. It didn't work.
>


In case you cannot active broken mirror that should work with --partial,
and you are using reasonable fresh version of lvm2 (bugs for 3 year old 
release will not gain a lot of attention) - please consider reporting
bug at bugzilla.redhat.com  (lvm2 component).

Zdenek

Re: [linux-lvm] Fun and games with mirroring

[Brassow Jonathan]

On May 22, 2012, at 2:59 PM, Vic wrote:

> 
>> LVM requires that for an LV to be activated while there are missing PVs,
>> the '--partial' flag must be used.  If the script running vgchange does
>> not include '--partial', your mirror won't be activated.
> 
> I tried that. It didn't work.

Pardon me, but I was only half way done with my explanation at that point.  The solution I was proposing to the problem began with "It seems to me that the solution might be...".  I described the difference between the two mirroring solutions so that it would be clear that solving this issue is probably not as simple as adding a '--partial' flag.

It would be very important to distinguish between the two mirror types and other LV types - linear for example.  The start-up scripts would probably be wise to automatically handle the case you are describing.  (I agree, you should be able to kill one of your devices while the machine is down and it should come up fine.)  However, if you encounter a linear LV - as Bryn mentioned - perhaps we should abort or force the user to confirm that they'd like to attempt to continue.

 brassow

Re: [linux-lvm] Fun and games with mirroring

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/22/2012 05:40 PM, Vic wrote:
>>> You can do it to a box that hasn't been set up with mdadm...
>> 
>> Sure but you will need LVM set up.
> 
> I always have.
> 
>> In that case, why not use LVM's snapshot capabilities?
> 
> Because that doesn't do what I want.
> 
> I'm not looking to have a LVM snapshot. I'm looking to have a
> duplicate disk. This allows me to experiment on one, whilst keeping
> the other safe.

That's exactly what snapshots do.

> Alternatively, it allows me to migrate from one disk to another
> without too much risk - before anyone points me at pvmove, I lost a
> filesystem when that fell over on a live machine once :-(

pvmove uses the exact same infrastructure as a mirrored LV; it just
constructs a temporary mirror over the source/destination extent
ranges, syncs it and breaks it leaving only the destination side
allocated so I wouldn't take past experience with pvmove as a reason
to prefer mirroring and splitting things.

> --splitmirrors would seem to do what I'm after *except* that it
> will leave the mirror unbootable without further attention (the
> machine won't boot with the wrong name on the LVs)

Sure but that's not an LVM problem - you're changing the name of
system LVs. That's always going to require some reconfiguration
(although in the snapshot case if you decide to keep the changes by
merging them back in the original LV name is preserved).

> This is true - but sorta misses the point; a LV mirror is not a LV
> mirror if it won't activate. Adding a mirror means that a faliure
> of *either* side renders the whole unusable.

Not really, it's just getting to where the problem is. LVM provides
tools to make a partial VG consistent again to allow activation of a
volume in the state you found. Right now this is not well integrated
with the initramfs activation - as you found it will break on a
partial volume group (any partial VG - mirroring is just a special
case of that where repair is more obvious and straightforward).

Fixing this properly may involve changes in both dracut and LVM.

> Well, we can call it Somebody Else's Problem, or perhaps LVM should
> be able to use a broken mirror - that's how every other mirror
> system works, and is generally the very reason for having a mirror
> at all. I consider this to be a bug within LVM.

Patches are welcome. LVM is designed not to activate a partial VG
unless the user specifies --partial. This is because for non-mirrored
LVs activating without all PVs present would leave holes in devices.
There is currently no logic to detect that the VG contains only
mirrored LVs and that sufficient PVs remain to allow safe conversion
to linear and activation. Implementing that would be a new feature.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+8pdYACgkQ6YSQoMYUY95QFgCg1aBWAC8CZ3UhaueXYrGcEYlK
zeYAn1dBCfkyv/DXGPMYueHnOEhn9dx0
=7uH9
-----END PGP SIGNATURE-----

Re: [linux-lvm] Fun and games with mirroring

[Les Mikesell]

On Wed, May 23, 2012 at 3:54 AM, Bryn M. Reeves <bmr@redhat.com> wrote:
>
>> I'm not looking to have a LVM snapshot. I'm looking to have a
>> duplicate disk. This allows me to experiment on one, whilst keeping
>> the other safe.
>
> That's exactly what snapshots do.

If your experiment involves reinstalling an OS that reformats one of
the drives, what will that do to your snapshot?

With MD raid1 you can just pull one of the mirrors and you are pretty
safe.   Although I'm beginning to like ReaR
(http://rear.sourceforge.net/) as a backup mechanism that doesn't
require shutting down to do the backup and can restore to bare metal
including reconstructing the underlying raid/lvm/filesystems.
Clonezilla can work too, but you have to shut down for the save and it
doesn't do raid (but on the plus side it can handle windows
partitions).

> Patches are welcome. LVM is designed not to activate a partial VG
> unless the user specifies --partial. This is because for non-mirrored
> LVs activating without all PVs present would leave holes in devices.

Wait, so LVM doesn't know if the volume it is starting is complete or
not in the case of mirrors and --partial just blindly starts anyway?
 That's one of those things where you have to ask what they were
thinking.

-- 
   Les Mikesell
      lesmikesell@gmail.com

Re: [linux-lvm] Fun and games with mirroring

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/23/2012 04:21 PM, Les Mikesell wrote:
> On Wed, May 23, 2012 at 3:54 AM, Bryn M. Reeves <bmr@redhat.com>
> wrote:
>> 
>>> I'm not looking to have a LVM snapshot. I'm looking to have a 
>>> duplicate disk. This allows me to experiment on one, whilst
>>> keeping the other safe.
>> 
>> That's exactly what snapshots do.
> 
> If your experiment involves reinstalling an OS that reformats one
> of the drives, what will that do to your snapshot?

If your experiment involves completely overwriting the device then
what was the point of snapshotting or imaging it in the first place?

You're just wasting time, I/O bandwidth and storage lifetime
shovelling data around for the purpose of immediately overwriting it.

Just do the experimental installation on your second disk and leave
the other one alone.

> With MD raid1 you can just pull one of the mirrors and you are
> pretty safe.   Although I'm beginning to like ReaR 
> (http://rear.sourceforge.net/) as a backup mechanism that doesn't 
> require shutting down to do the backup and can restore to bare
> metal including reconstructing the underlying
> raid/lvm/filesystems. Clonezilla can work too, but you have to shut
> down for the save and it doesn't do raid (but on the plus side it
> can handle windows partitions).
> 
>> Patches are welcome. LVM is designed not to activate a partial
>> VG unless the user specifies --partial. This is because for
>> non-mirrored LVs activating without all PVs present would leave
>> holes in devices.
> 
> Wait, so LVM doesn't know if the volume it is starting is complete
> or not in the case of mirrors and --partial just blindly starts
> anyway?

Wait, what? I'm not sure how you got to that from the paragraph you
quoted. It's not what was stated and is not correct.

The --partial switch is *designed* for recovery scenarios where you
have one or more missing devices. The one and only thing --partial
does is to tell LVM to activate even though devices are missing and to
fill any gaps as best it can.

It's not "blindly" starting things - it is doing what the
administrator told it to do: attempt to activate a VG with missing
devices substituting the error target (or whatever the admin
configured as activation{missing_stripe_filler}) for any segments
allocated on missing devices.

> LVM doesn't know if the volume it is starting is complete or not in
> the case of mirrors

No. That's not what I said. I said that LVM will not start a VG that
has missing PVs unless --partial is specified because it could leave
holes in LVs (specifically non-mirrored LVs).

> That's one of those things where you have to ask what they were 
> thinking.

Actually, I have to ask that in regards to your mail ;-)

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+9A/AACgkQ6YSQoMYUY96h9gCfcP0XsBBnMMOFN+Lh+I5Mt6Bt
QNoAoMERSvumSpDryzvU8dqk4hdxtxMm
=kdRP
-----END PGP SIGNATURE-----

Re: [linux-lvm] Fun and games with mirroring

[Les Mikesell]

On Wed, May 23, 2012 at 10:36 AM, Bryn M. Reeves <bmr@redhat.com> wrote:
>>>
>> If your experiment involves reinstalling an OS that reformats one
>> of the drives, what will that do to your snapshot?
>
> If your experiment involves completely overwriting the device then
> what was the point of snapshotting or imaging it in the first place?

So you have an instant 'undo'.

> You're just wasting time, I/O bandwidth and storage lifetime
> shovelling data around for the purpose of immediately overwriting it.

By definition you don't know the result an experiment.  You may want
the new replacement, you may want to recover.

> Just do the experimental installation on your second disk and leave
> the other one alone.

That's a reasonable option too...  But it might not be the way you
want to end up.

>
>>> Patches are welcome. LVM is designed not to activate a partial
>>> VG unless the user specifies --partial. This is because for
>>> non-mirrored LVs activating without all PVs present would leave
>>> holes in devices.
>>
>> Wait, so LVM doesn't know if the volume it is starting is complete
>> or not in the case of mirrors and --partial just blindly starts
>> anyway?
>
> Wait, what? I'm not sure how you got to that from the paragraph you
> quoted. It's not what was stated and is not correct.
>
> The --partial switch is *designed* for recovery scenarios where you
> have one or more missing devices. The one and only thing --partial
> does is to tell LVM to activate even though devices are missing and to
> fill any gaps as best it can.
>
> It's not "blindly" starting things - it is doing what the
> administrator told it to do: attempt to activate a VG with missing
> devices substituting the error target (or whatever the admin
> configured as activation{missing_stripe_filler}) for any segments
> allocated on missing devices.

How is it not blind if it does not in fact know that a mirror is or
isn't complete?   Or at least if it doesn't give me the option to say
start only if it has at least one complete copy of the data?

>> LVM doesn't know if the volume it is starting is complete or not in
>> the case of mirrors
>
> No. That's not what I said. I said that LVM will not start a VG that
> has missing PVs unless --partial is specified because it could leave
> holes in LVs (specifically non-mirrored LVs).
>
>> That's one of those things where you have to ask what they were
>> thinking.
>
> Actually, I have to ask that in regards to your mail ;-)

If I have an MD device it will know whether it is reasonable to
assemble or not.   A missing member in raid1 is still reasonable to
run since it still has a complete copy of the filesystems on it.
Where is the 'start if reasonable' concept in LVM?    That is, if it
won't start automatically with a missing mirror, it seems like I have
to tell it to start even if it does not have a complete image - which
doesn't seem reasonable.

-- 
   Les Mikesell
     lesmikesell@gmail.com

Re: [linux-lvm] Fun and games with mirroring

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/23/2012 06:03 PM, Les Mikesell wrote:
> On Wed, May 23, 2012 at 10:36 AM, Bryn M. Reeves <bmr@redhat.com>
> wrote:
>> If your experiment involves completely overwriting the device
>> then what was the point of snapshotting or imaging it in the
>> first place?
> 
> So you have an instant 'undo'.

You already have one: switch to the second drive.

>> You're just wasting time, I/O bandwidth and storage lifetime 
>> shovelling data around for the purpose of immediately overwriting
>> it.
> 
> By definition you don't know the result an experiment.  You may
> want the new replacement, you may want to recover.

In this case you know very well the result; everything you just backed
up has been overwritten (at least to the point of being very hard to
recover). There is no point in doing that.

>> Just do the experimental installation on your second disk and
>> leave the other one alone.
> 
> That's a reasonable option too...  But it might not be the way you 
> want to end up.

You have two disks. You want to put something new on one of them.
Explain to me again why copying all the data from the first to the
second before completely overwriting it with a new install does
anything useful?

> How is it not blind if it does not in fact know that a mirror is
> or isn't complete?   Or at least if it doesn't give me the option
> to say start only if it has at least one complete copy of the
> data?

Again: that is not what I said. Please read again. The --partial
option is DESIGNED to activate devices with bits missing. If you use
it you, the administrator are asking for that to happen. There is
nothing blind about that.

The *default* is to refuse to ever activate any VG of any kind that
has missing devices.

In the case of mirrors vgchange /could/ figure out that all the
devices for one whole side are there and activate a consistent device
/without/ --partial but as Vic noticed that is not yet implemented.

I don't see how you construe from that that vgchange activates devices
that it does not know are complete let alone how you get to it not
giving you the option to avoid that. It is the default.

> If I have an MD device it will know whether it is reasonable to 
> assemble or not.   A missing member in raid1 is still reasonable
> to run since it still has a complete copy of the filesystems on
> it. Where is the 'start if reasonable' concept in LVM?    That is,
> if it won't start automatically with a missing mirror, it seems
> like I have to tell it to start even if it does not have a complete
> image - which doesn't seem reasonable.

Nobody is arguing that the handling of mirrored LVs cannot be
improved; I'd like to see vgchange apply the fault policy during
activation and tidy up automatically but this is not implemented today.

Right now the tools give the administrator a means to make the VG
consistent again and to reinstate mirroring for the LV on another
device if desired.

How the administrator chooses to use or not use those tools while
waiting for new features to arrive is entirely his or her own choice.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+9HUcACgkQ6YSQoMYUY961EACgzWdASkFnxjqAWD8HhOceN2EA
y5kAnR3BWTBQN5VGXgdS0+ReJM+T8Cqw
=2uxE
-----END PGP SIGNATURE-----

Re: [linux-lvm] Fun and games with mirroring

[Les Mikesell]

On Wed, May 23, 2012 at 12:24 PM, Bryn M. Reeves <bmr@redhat.com> wrote:
>
>>> Just do the experimental installation on your second disk and
>>> leave the other one alone.
>>
>> That's a reasonable option too... �But it might not be the way you
>> want to end up.
>
> You have two disks. You want to put something new on one of them.
> Explain to me again why copying all the data from the first to the
> second before completely overwriting it with a new install does
> anything useful?

I normally want my data on two disks all of the time (and have a large
pile of failed drives waiting to be destroyed to back up the reasoning
there...).  But I might tolerate only one fresh copy for the duration
of testing a possible replacement OS/filesystem.  So in the 'break the
mirror' case, the copy is already there, and in an instantly usable
form if I want it back.

The unfortunate part is that most linux distributions don't allow
installation on a 'broken mirror' or have a way to convert from
non-raid to raid after the install the way you can on a windows server
- or if there is, I haven't found it.  So even if you like the new
replacement that you tried on the single-disk install, you have to
throw it away and re-install on raid in the end.

>> How is it not blind if it does not in fact know that a mirror is
>> or isn't complete? � Or at least if it doesn't give me the option
>> to say start only if it has at least one complete copy of the
>> data?
>
> Again: that is not what I said. Please read again. The --partial
> option is DESIGNED to activate devices with bits missing. If you use
> it you, the administrator are asking for that to happen. There is
> nothing blind about that.

So what is the command to start in the case where devices are missing,
but no bits?  That is, under conditions where it is reasonable to
start.

>
> I don't see how you construe from that that vgchange activates devices
> that it does not know are complete let alone how you get to it not
> giving you the option to avoid that. It is the default.

I construed that because you didn't say it is possible.  Half of a
mirror is complete, more or less by the definition of mirroring.  But
the device should know that, not the administrator.   The MD raid1
case is usually pretty simple, but I thought LVs could be spread
across devices in ways that the administrator (much less an operator)
of the machine might not know if a complete half of the mirror is
present or not or whether starting with --partial is a sane choice.

> How the administrator chooses to use or not use those tools while
> waiting for new features to arrive is entirely his or her own choice.

Personally, I've generally chosen not to use LVM at all most of the
time because it hasn't seemed to offer anything I'd want in a
real-world situation and adds unnecessary overhead, but maybe I'm just
missing something.  But it is extra work during installs to do my own
layout so I keep reading this list for something that might change my
mind...

-- 
  Les Mikesell
     lesmikesell@gmail.com

Re: [linux-lvm] Fun and games with mirroring

[Stuart D Gathman]

On May 23, Les Mikesell transmitted in part:

> I normally want my data on two disks all of the time (and have a large
> pile of failed drives waiting to be destroyed to back up the reasoning
> there...).  But I might tolerate only one fresh copy for the duration
> of testing a possible replacement OS/filesystem.  So in the 'break the
> mirror' case, the copy is already there, and in an instantly usable
> form if I want it back.
>
> The unfortunate part is that most linux distributions don't allow
> installation on a 'broken mirror' or have a way to convert from
> non-raid to raid after the install the way you can on a windows server
> - or if there is, I haven't found it.  So even if you like the new
> replacement that you tried on the single-disk install, you have to
> throw it away and re-install on raid in the end.

I do this all the time - but with MD raid.  Use custom install, and
create RAID1 MD devices in the install with only 1 leg.  You can add
mirrors later with mdadm.  I also have customers pull out a disk and
send it to me so I can quickly clone their system (while they continue
to operate) on new hardware.  I also routinely add an MD mirror on a new
drive to migrate a PV to new hardware.

It is only the LVM mirroring that isn't doing what you want.

My only complaint with the MD driver is that it doesn't divide up drives
into small partitions.  I often end up creating 2 or 3 partitions on a drive,
and mirroring each.  This allows later distributing them over drives of
unequal size.  E.g., you start out with 2x512G, then later add a 1T,
which you split into 2 partitions big enough to mirror your original
drives, thus doubling your mirrored space with 3 drives.  This kind
of thing would be automatic with a more complete feature set in LVM
mirroring.

It could make sense, however, to add such features to the MD driver,
and keep LVM focused on LVs (and snapshot pools and such) rather than
mirroring.  And that seems to be the way MD + LVM is going now.

Re: [linux-lvm] Fun and games with mirroring

[Les Mikesell]

On Wed, May 23, 2012 at 1:32 PM, Stuart D Gathman <stuart@bmsi.com> wrote:
> >
>> The unfortunate part is that most linux distributions don't allow
>> installation on a 'broken mirror' or have a way to convert from
>> non-raid to raid after the install the way you can on a windows server
>> - or if there is, I haven't found it. �So even if you like the new
>> replacement that you tried on the single-disk install, you have to
>> throw it away and re-install on raid in the end.
>
>
> I do this all the time - but with MD raid. �Use custom install, and
> create RAID1 MD devices in the install with only 1 leg. �You can add
> mirrors later with mdadm.

I've done that manually in a separate virtual terminal during the
install, partly because the installer likes to re-arrange the order of
the partitions on the disk when you define them there and partly
because it didn't occur to me that the installer would accept a raid1
with only one member.  Maybe I'll give it another try.  But with the
installer willing to align things on 2M boundaries these days you'd
think there would be a way to set things up so you could back in
whatever MD needs later.

> My only complaint with the MD driver is that it doesn't divide up drives
> into small partitions. �I often end up creating 2 or 3 partitions on a
> drive,
> and mirroring each.

I thought these days you could raid the whole disk and put partitions
on the raid, although I've never done it that way.  I have 2
complaints about MD.  One is that it doesn't handle errors/retries on
members very well - it will kick members that are perfectly fine
running standalone or as the last remaining member, and the other is
that partitions over 2TB need a new format that the kernel doesn't
auto-detect.  Much of the reason I liked MD in the first place was its
ability to assemble things correctly at boot regardless of the drive's
physical attachment or location.

-- 
  Les Mikesell
    lesmikesell@gmail.com

Re: [linux-lvm] Fun and games with mirroring

[Stuart D Gathman]

On 05/23/2012 03:07 PM, Les Mikesell expounded in part:
> I thought these days you could raid the whole disk and put partitions 
> on the raid, although I've never done it that way. I have 2 complaints 
> about MD. One is that it doesn't handle errors/retries on members very 
> well - it will kick members that are perfectly fine running standalone 
> or as the last remaining member, and the other is that partitions over 
> 2TB need a new format that the kernel doesn't auto-detect. Much of the 
> reason I liked MD in the first place was its ability to assemble 
> things correctly at boot regardless of the drive's physical attachment 
> or location. 
You can raid the whole disk.  But having smaller partitions makes it 
easier to distribute them over odd numbers of disks.  Back when I ran 
AIX, you could add a disk of any size to the VG, and mirrored LVs would 
allocate LPs from physically separate PPs.   You never had to worry 
about matching disk sizes.  With MD, you can also add odd disk sizes and 
fully use them, but you have to do it manually, and more solutions are 
available with smaller partitions.  (But having too many is too tedious.)

Re: [linux-lvm] Fun and games with mirroring

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 05/22/2012 10:50 AM, Vic would write:
> p.s. The reason I want to do this with LVM mirrors rather than MD RAID is
> that this is potentially a very useful way of grabbing a backup snapshot
> of a machine prior to some risky operation - it allows me to "fork" a
> machine onto two sets of media. But if it leaves the result unbootable,
> that's something of a problem...
>
LVM RAID is mainly used for pvmove at the moment.  LVM has a ways to go 
to match ZFS or AIX in LVM mirroring features.   MD RAID takes a 
snapshot just fine.  Add the mirror with mdadm, sync it, and "fail" the 
mirror and remove it before physically removing the drive.  I can see 
the issue with multiple PVs, however.  There is no atomic way to fail 
multiple mirrors at once.

Re: [linux-lvm] Fun and games with mirroring

[tariq wali]

[-- Attachment #1: Type: text/plain, Size: 1934 bytes --]

if I may ask an elementary question please :) I normally take a snapshot on
lvm partition offcourse that contains MySQL data say /data and then mount
it as ro and then use the ancient tar/scp to copy it elsewhere . Could
anyone please correct me if there is a better way to do it ? like can i
take  a snapshot of /data and transfer the entire volume to another system
( i mean duplicate/mirror it ) ?

Also

sometime /data contains data that I really don't need to be in the snapshot
but the way i take snapshot it pretty much creates a mirror .. I have
always wondered how snapshot thing works because when i take snapshot I
continue to use my original /data and continue to do IO on it while
snapshot is mounted :)



On Tue, May 22, 2012 at 11:27 PM, Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 05/22/2012 10:50 AM, Vic would write:
>
>> p.s. The reason I want to do this with LVM mirrors rather than MD RAID is
>> that this is potentially a very useful way of grabbing a backup snapshot
>> of a machine prior to some risky operation - it allows me to "fork" a
>> machine onto two sets of media. But if it leaves the result unbootable,
>> that's something of a problem...
>>
>>  LVM RAID is mainly used for pvmove at the moment.  LVM has a ways to go
> to match ZFS or AIX in LVM mirroring features.   MD RAID takes a snapshot
> just fine.  Add the mirror with mdadm, sync it, and "fail" the mirror and
> remove it before physically removing the drive.  I can see the issue with
> multiple PVs, however.  There is no atomic way to fail multiple mirrors at
> once.
>
>
>
> ______________________________**_________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/**mailman/listinfo/linux-lvm<https://www.redhat.com/mailman/listinfo/linux-lvm>
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-**HOWTO/<http://tldp.org/HOWTO/LVM-HOWTO/>
>



-- 
*Tariq Wali.*

[-- Attachment #2: Type: text/html, Size: 2842 bytes --]

Re: [linux-lvm] Fun and games with mirroring

[Stuart D Gathman]

On 05/22/2012 02:39 PM, tariq wali expounded in part:
> if I may ask an elementary question please :) I normally take a 
> snapshot on lvm partition offcourse that contains MySQL data say /data 
> and then mount it as ro and then use the ancient tar/scp to copy it 
> elsewhere . Could anyone please correct me if there is a better way to 
> do it ? like can i take  a snapshot of /data and transfer the entire 
> volume to another system ( i mean duplicate/mirror it ) ?
Yes
>
> sometime /data contains data that I really don't need to be in the 
> snapshot but the way i take snapshot it pretty much creates a mirror 
> .. I have always wondered how snapshot thing works because when i take 
> snapshot I continue to use my original /data and continue to do IO on 
> it while snapshot is mounted :)
>
Writes to the original volume cause the original data to be copied to 
the snapshot volume first.  Reads from the snapshot check for the block 
in the snapshot volume, and if not there read it from the original 
volume.  Thus, each first time logical write to a block on the original 
volume results in a read,write,write - and performance could tank if you 
are doing too many of these.