[linux-lvm] Auto PV scan w/ LVM

[linux-lvm] Auto PV scan w/ LVM

[David Coulson]

I've got a couple of Clariion disk frames on which we performing cloning 
of LUNs for backups using HP-UX systems. Basically box A has a LUN or 
two, we clone it, and present it to box B - It mounts it, backs it up, 
then umounts it and we restart the cloning process... Nothing magical.

I'm working towards replicating this process for Linux systems, and keep 
hitting a road block. The clone LUN is always presented to the backup 
server, however while it is syncing with the source LUN it is not 
available for IO directly. In this state multipathd marks both paths as 
failed, which I'm okay with:

[root@rhtsutility02 ~]# multipath -ll 3600601608d211a00f4e21b978136df11
3600601608d211a00f4e21b978136df11
[size=20 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\_ round-robin 0 [prio=1][enabled]
  \_ 15:0:0:12 sdad 65:208 [failed][ready]
\_ round-robin 0 [enabled]
  \_ 17:0:0:12 sdac 65:192 [failed][ready]

Unfortunately, LVM seems to automatically pick up on PVs within the 
cloned LUN as soon as multipathd grabs it, so as soon as I do a 'vgscan' 
it hangs until I break the clone and present the cloned LUN to my backup 
box.

So far I've basically excluded these cloned LUNs from lvm with a filter, 
however I do need to enable the VGs on them and mount LVs for backups. 
I'm not sure if switching out the filter all the time is the best way to 
go, but it doesn't look like LVM2's version of vgimport allows you to 
specify a PV device.

Anyone had to deal with this before, or have an idea of a good way to 
deal with it? I agree it's a messy situation, but EMC is good at that :-)

Re: [linux-lvm] Auto PV scan w/ LVM

[Mike Snitzer]

On Fri, Mar 26 2010 at 10:16am -0400,
David Coulson <david@davidcoulson.net> wrote:

> I've got a couple of Clariion disk frames on which we performing
> cloning of LUNs for backups using HP-UX systems. Basically box A has
> a LUN or two, we clone it, and present it to box B - It mounts it,
> backs it up, then umounts it and we restart the cloning process...
> Nothing magical.
> 
> I'm working towards replicating this process for Linux systems, and
> keep hitting a road block. The clone LUN is always presented to the
> backup server, however while it is syncing with the source LUN it is
> not available for IO directly. In this state multipathd marks both
> paths as failed, which I'm okay with:
> 
> [root@rhtsutility02 ~]# multipath -ll 3600601608d211a00f4e21b978136df11
> 3600601608d211a00f4e21b978136df11
> [size=20 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
> \_ round-robin 0 [prio=1][enabled]
>  \_ 15:0:0:12 sdad 65:208 [failed][ready]
> \_ round-robin 0 [enabled]
>  \_ 17:0:0:12 sdac 65:192 [failed][ready]
> 
> Unfortunately, LVM seems to automatically pick up on PVs within the
> cloned LUN as soon as multipathd grabs it, so as soon as I do a
> 'vgscan' it hangs until I break the clone and present the cloned LUN
> to my backup box.
> 
> So far I've basically excluded these cloned LUNs from lvm with a
> filter, however I do need to enable the VGs on them and mount LVs
> for backups. I'm not sure if switching out the filter all the time
> is the best way to go, but it doesn't look like LVM2's version of
> vgimport allows you to specify a PV device.
> 
> Anyone had to deal with this before, or have an idea of a good way
> to deal with it? I agree it's a messy situation, but EMC is good at
> that :-)

I'm not crystal clear on your use-case (e.g. the mechanics and
constraints of when the clone is presented to the backup, etc).

But the 'vgimportclone' command will enable you to make the cloned LUNs
distinct from the originals.  Does this help you?

Mike

Re: [linux-lvm] Auto PV scan w/ LVM

[David Coulson]

On 3/26/2010 12:18 PM, Mike Snitzer wrote:
> I'm not crystal clear on your use-case (e.g. the mechanics and
> constraints of when the clone is presented to the backup, etc).
>    

The process is basically this:

* Clone in 'sync' state so it is replicating block changes from the 
source. The clone LUN is still visible to the backup server, however 
both paths are 'failed' to multipathd
* We fracture/break the clone which makes the clone LUN available to the 
backup server for IO
* We mount LVs from the clone LUN on the backup server, run our backup 
job, unmount it
* vgexport the clone LUN VG and remove it from our multipath list with 
'multipath -f'
* Tell the SAN to start syncing the clone to source again until we need 
to break it for another backup.

I basically need to find a nice way to make LVM ignore the clone disk on 
the backup system until I am ready to make use of it for a backup. 
multipathd sees it as failed during this time, but vgscan/pvscan, etc 
all hang. Maybe that is normal.

> But the 'vgimportclone' command will enable you to make the cloned LUNs
> distinct from the originals.  Does this help you?
>    
The clone gets mounted on a different box to the source, so it's not 
important to me. I don't have a vgimportclone command on RHEL4, unless I 
need to install some additional packages.

David

Re: [linux-lvm] Auto PV scan w/ LVM

[Mike Snitzer]

On Fri, Mar 26 2010 at 12:29pm -0400,
David Coulson <david@davidcoulson.net> wrote:

> On 3/26/2010 12:18 PM, Mike Snitzer wrote:
> >I'm not crystal clear on your use-case (e.g. the mechanics and
> >constraints of when the clone is presented to the backup, etc).
> 
> The process is basically this:
> 
> * Clone in 'sync' state so it is replicating block changes from the
> source. The clone LUN is still visible to the backup server, however
> both paths are 'failed' to multipathd
> * We fracture/break the clone which makes the clone LUN available to
> the backup server for IO
> * We mount LVs from the clone LUN on the backup server, run our
> backup job, unmount it
> * vgexport the clone LUN VG and remove it from our multipath list
> with 'multipath -f'
> * Tell the SAN to start syncing the clone to source again until we
> need to break it for another backup.
> 
> I basically need to find a nice way to make LVM ignore the clone
> disk on the backup system until I am ready to make use of it for a
> backup. multipathd sees it as failed during this time, but
> vgscan/pvscan, etc all hang. Maybe that is normal.

Are you using multipath's 'queue_if_no_path' (assuming RHEL4 has it)?
Could be LVM's IO to the mpath devices will queue (block) until the
point when you "break the clone" (which makes the paths(s) available).

You could get very specific with the multipath configuration for these
clone LUNs to fail IO immediately (in theory).  But in practice this is
a delicate task that would require special EMC LUN understanding of how
to differentiate between a clone LUN that is actively sync'ing and just
some other LUN.  It also would be challenging to manage these changes at
runtime (might require multipathd being restarted with updated config
based on WWID, etc).
 
> I basically need to find a nice way to make LVM ignore the clone
> disk on the backup system until I am ready to make use of it for a
> backup. multipathd sees it as failed during this time, but
> vgscan/pvscan, etc all hang. Maybe that is normal.

Are you aware of LVM_SYSTEM_DIR environment variable?  It could be that
you have a default lvm.conf that rejects (via filters) the clone LUNs.
But when you want to be able to access the clone LUNs for backup you
could have a custom lvm.conf that allows (via filters) these clone
LUNs.  So each shell you'd be doing your backup work in would point to
this custom lvm.conf that lives in $LVM_SYSTEM_DIR.

<write custom filters allowing clone LUNs to /etc/backup_lvm/lvm.conf>

export LVM_SYSTEM_DIR=/etc/backup_lvm/
<do backup commands>
unset LVM_SYSTEM_DIR

> >But the 'vgimportclone' command will enable you to make the cloned LUNs
> >distinct from the originals.  Does this help you?
> The clone gets mounted on a different box to the source, so it's not
> important to me. I don't have a vgimportclone command on RHEL4,
> unless I need to install some additional packages.

vgimportclone is not packaged in RHEL4's lvm2.  But vgimportclone
wouldn't help you anyway; thanks for clarifying.

Mike

Re: [linux-lvm] Auto PV scan w/ LVM

[David Coulson]

On 3/26/2010 2:16 PM, Mike Snitzer wrote:
> Are you using multipath's 'queue_if_no_path' (assuming RHEL4 has it)?
> Could be LVM's IO to the mpath devices will queue (block) until the
> point when you "break the clone" (which makes the paths(s) available).
>
This seems to be a pretty good solution to the problem - When I set it 
to fail_if_no_path via dmsetup, it seemed to work nicely. Any idea how 
to set this within a multipaths { multipath { wwid xxxx }} config?