[linux-lvm] Re: LVM on dmraid breakage

[linux-lvm] Re: LVM on dmraid breakage

[Luca Berra]

On Wed, Aug 01, 2007 at 04:19:59PM -0400, Phillip Susi wrote:
>I am investigating a bug report involving the combination of lvm and 
>dmraid, and it seems to me that the problem is that lvm is detecting the 
>underlying partition on the primary disk and accessing it directly 
>rather than going through the raid device made by dmraid.  I think that 
>the problem is there is no facility for dmraid to "claim" the physical 
>disk so that lvm does not look at it, and also to have lvm scan the raid 
>device for physical volumes.
>
>Does anyone have any thoughts on how this might be accomplished?

Two things come to mind
1) dmraid should use ioctl BLKPG_DEL_PARTITION, to clean up any eventual
partition table on component devices

2) lvm tool filter could be modified to check if a device has been
already claimed by device-mapper.

something along the lines of:
ioctl DM_LIST_DEVICES
while (...)
  ioctl DM_TABLE_DEPS

the above would probably require a cache.


i believe that manual use of partx or lvm filters is sub-optimal
L.



-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: LVM on dmraid breakage

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luca Berra wrote:
> Two things come to mind
> 1) dmraid should use ioctl BLKPG_DEL_PARTITION, to clean up any eventual
> partition table on component devices
> 
> 2) lvm tool filter could be modified to check if a device has been
> already claimed by device-mapper.
> 

But device-mapper already opens devices exclusively when a target claims
them.

The problem here sounds more like it's down to ordering; LVM2 is being
given a chance to scan the devices & build LVs / claim devices before
dmraid gets a look at them.


Kind regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGsbXh6YSQoMYUY94RAp5aAKDHrjX/LIoM5XPkeD/3hkd+oQ9vHgCZAXHS
Ykme+1gwvYuRat0fZwBALJs=
=mIsK
-----END PGP SIGNATURE-----

Re: [linux-lvm] Re: LVM on dmraid breakage

[Luca Berra]

On Thu, Aug 02, 2007 at 11:45:53AM +0100, Bryn M. Reeves wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Luca Berra wrote:
>> Two things come to mind
>> 1) dmraid should use ioctl BLKPG_DEL_PARTITION, to clean up any eventual
>> partition table on component devices
>> 
>> 2) lvm tool filter could be modified to check if a device has been
>> already claimed by device-mapper.
>> 
>
>But device-mapper already opens devices exclusively when a target claims
>them.

No, it does not,
sorry

[root@localhost ~]# dmsetup create foobar
0 281329664 linear /dev/md2 0
[root@localhost ~]# dmsetup table
foobar: 0 281329664 linear 9:2 0
[root@localhost ~]# dmsetup create foobar1
0 281329664 linear /dev/md2 0
[root@localhost ~]# dmsetup create foobar2
0 281329664 linear /dev/md2 0
[root@localhost ~]# dmsetup create foobar2
0 281329664 linear /dev/md2 0
[root@localhost ~]# dmsetup table
foobar3: 0 281329664 linear 9:2 0
foobar2: 0 281329664 linear 9:2 0
foobar1: 0 281329664 linear 9:2 0
foobar: 0 281329664 linear 9:2 0

L.
-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[David Robinson]

> The problem is that lvm is looking at the physical disks at all.  It 
> needs to scan the dmraid device, not the underlying disks.

If you need LVM to only scan certain devices you should modify the 
filter line in lvm.conf.

--Dave

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Fri, Aug 03, 2007 at 10:40:45AM +1000, David Robinson wrote:
> If you need LVM to only scan certain devices you should modify the 
> filter line in lvm.conf.
 
Yes, this is the current approach: a well-configured system would be
actively modifying the filters (with the help of udev) both on boot and
if device configurations change subsequently.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Thu, Aug 02, 2007 at 05:50:51PM -0400, Phillip Susi wrote:
> The problem is that lvm is looking at the physical disks at all.  It 
> needs to scan the dmraid device, not the underlying disks.
 
Indeed.  Please send the evidence showing where it is going wrong so
we can fix it.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Wed, Aug 08, 2007 at 05:04:12PM -0400, Phillip Susi wrote:
> https://bugs.launchpad.net/ubuntu/+source/dmraid/+bug/129285
 
There's something not quite right there:
  Why doesn't the 'lvdisplay' issue a message about the duplicate PV?
  - run 'vgscan -vvvv' to see what lvm2 is really doing with all the
    relevant devices

  What lvm2 version? 
   - could be old and not containing the code to do the detection
     correctly?

  Are there non-upstream-default config settings that had the effect of turning
off the detection?  
   - see 'lvm dumpconfig' output or /etc/lvm/lvm.conf plus any compile-time
     default changes


Standard procedure these days is to ask people to run 'lvmdump -a'
(scripts/lvm_dump.sh in the source tree) to capture the relevant information
for examination.

> This is do to the fact that the lvm tools open /dev/sd* and look for lvm 
> partitions, which it finds.
 
But I don't yet see why it didn't also open the other devices, write a warning
message about duplicate IDs, and do the right thing and give them
precedence.
 
Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Luca Berra]

On Thu, Aug 09, 2007 at 01:27:34PM -0400, Phillip Susi wrote:
>Alasdair G Kergon wrote:
>>But I don't yet see why it didn't also open the other devices, write a 
>>warning
>>message about duplicate IDs, and do the right thing and give them
>>precedence.
>
>Why would it open the other devices?  LVM does not scan /dev/mapper/*, 
>only /dev/sd*.  Since it does not scan the /dev/mapper devices for pvs, 
>it sees no duplicates and can not "do the right thing".

LVM scans the whole /dev directory, unless you configure it otherwise.

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Thu, Aug 09, 2007 at 01:27:34PM -0400, Phillip Susi wrote:
> Why would it open the other devices?  LVM does not scan /dev/mapper/*, 
> only /dev/sd*.

Upstream lvm2 has done for a long time (since 2.02.00 with quite a few
fixes and tweaks after that, and one known problem still outstanding).
I can't speak for any customisation done in ubuntu or on the machine
with the problem - hence my request for full version and configuration
information.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Thu, Aug 02, 2007 at 08:50:12AM +0200, Luca Berra wrote:
> 2) lvm tool filter could be modified to check if a device has been
> already claimed by device-mapper.
> something along the lines of:
> ioctl DM_LIST_DEVICES
> while (...)
>  ioctl DM_TABLE_DEPS

There are already some checks in the code, but they are have not been completed.

But a dm device should already take precedence over a scsi device.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Luca Berra]

On Thu, Aug 02, 2007 at 06:04:20PM -0400, Phillip Susi wrote:
>Luca Berra wrote:
>>Two things come to mind
>>1) dmraid should use ioctl BLKPG_DEL_PARTITION, to clean up any eventual
>>partition table on component devices
>
>How will this play out in terms of udev plug events?  Can it be relied 
>on that udev will process the disk first, before the partitions?  If the 
>partitions are deleted during the processing of the disk, does udev 
>still try to process their add events?  Will it then process the delete 
>events?  If so then this is problematic.
I really have no clue, fortunately dmraid is usually used for boot
disks, so it should be started before udev has a chance of messing with
devices. btw, md already does this, and redhat's initrd does this as
well.
The real solution to this is so simple.
Just ditch the partition detection code from the kernel, and move it to
userspace where it belongs.
For the time being use BLKPG_DEL_PARTITION, to undo what the kernel
should not have done.

>>2) lvm tool filter could be modified to check if a device has been
>>already claimed by device-mapper.
>>
>>something along the lines of:
>>ioctl DM_LIST_DEVICES
>>while (...)
>>  ioctl DM_TABLE_DEPS
>>
>>the above would probably require a cache.
>
>I'd rather not see it that tightly coupled to dm.  I was thinking 
>perhaps of something involving udev attributes so that udev would know 
>not to run pvscan on that device.  Though it looks like pvscan also 
>needs reworked so it plays nice with udev, being called to scan a given 
>device as detected rather than looking for all well known physical 
>device names in /dev.
i'd rather not see it coupled with udev :P
maybe i am limited, but i really fail to see how an event driven model
could be at all useful in this cases, and i am really convinced that the
effort needed to make i work is too high compared to possible benefits.
The biggest question being: How do you know you have scanned all
possible PV for a given volume group, and you have to activate it.
Anyway i still think my proposal is sensible and adapts to the current
paradigm of lvm2.

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Fri, Aug 03, 2007 at 10:11:34AM +0200, Luca Berra wrote:
> i'd rather not see it coupled with udev :P
> maybe i am limited, but i really fail to see how an event driven model
> could be at all useful in this cases, and i am really convinced that the
> effort needed to make i work is too high compared to possible benefits.

This is out of our hands - people are already experimenting with this
and lvm2 needs to support it with enthusiasm!

But that does not mean we'll stop supporting the existing mechanisms, as
it'll just become a configuration option (I suspect compile-time rather
than run-time).

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Luca Berra]

On Fri, Aug 03, 2007 at 02:10:02PM -0400, Phillip Susi wrote:
>I agree with moving the partition detection code to user space, but 
>trying to undo it after the fact doesn't help because udev is already 
>processing the add events.  Also you do not need to remove the 
>partitions so long as pvscan understands that it shouldn't be using them.
which is the modification i proposed to lvm tools, isn't it?


>Udev is supposed to be the new model for enumerating devices and 
i know that, and i will withdraw from this discussion, since it might
get to an useless flame war.

Is there any technical reason for not having lvm tools filter out devices that
are used by device mapper?

besides dmraid, think of multipath.

Regards,
L.


-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Luca Berra]

On Fri, Aug 03, 2007 at 05:30:00PM -0400, Phillip Susi wrote:
>Luca Berra wrote:
>>On Fri, Aug 03, 2007 at 02:10:02PM -0400, Phillip Susi wrote:
>>>I agree with moving the partition detection code to user space, but 
>>>trying to undo it after the fact doesn't help because udev is already 
>>>processing the add events.  Also you do not need to remove the 
>>>partitions so long as pvscan understands that it shouldn't be using them.
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>which is the modification i proposed to lvm tools, isn't it?
>
>You suggested deleting the partition table after it has already been 
>detected.  I am saying that while I agree in principal that partition 
I was referring to the modification to lvm2, not the one to dmraid.

>detection should be moved out of the kernel, for now, it is in there and 
>deleting them after they have already been detected doesn't help matters 
>because pvscan may already be running on them.
Actually i am not worried too much about vgscan/pvscan, i am much more
worried for mount, swapon, one particular piece of crap called
hibernate-cleanup.sh etc.
besides, i am positive that users get confused having both /dev/sda1 and
/dev/mapper/via_aggehhahyue1

>>>Udev is supposed to be the new model for enumerating devices and 
>>i know that, and i will withdraw from this discussion, since it might
>>get to an useless flame war.
>>
>>Is there any technical reason for not having lvm tools filter out 
>>devices that
>>are used by device mapper?
>>
>>besides dmraid, think of multipath.
>
>None that I can see at the moment, but that doesn't mean there isn't 
>one, or won't be one in the future.  The other problem is that there are 
the above is called FUD

>likely other factors besides being used already as a dm target that 
>might give reason for lvm to not scan the volume.  These kind of policy 
probably, we strive to be perfect, but we still have a long way to
go...

>decisions seem like they should be made by udev rather than hard coded 
>into lvm.  If the admin wants a policy where lvm should look at volumes, 
>or indeed, maybe only certain volumes, that already happen to be dm 
>targets, he should be able to do that.  Likewise, there may be some 
udev is not the only thing on earth that wants to activate a volume
group. what if i wanted to do it manually?

>other reason to not look at a disk for lvm pvs.  Editing a conf file to 
>specify a filter list of devices by name is all well and good for a 
>static system, but it does not play well in the modern udev managed plug 
>and play world.
whoever wrote about editing the conf file?
i wrote about detecting that a device is already in user by
device-mapper and skipping that.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Luca Berra]

On Mon, Aug 06, 2007 at 10:45:02AM -0400, Phillip Susi wrote:
>>udev is not the only thing on earth that wants to activate a volume
>>group. what if i wanted to do it manually?
>
>Then you do so manually.  This discussion is about what the system does 
>automatically.
And by activating a vg manually i don't benefit from the filter? why?

What about if it is a ha-cluster suite that needs to activate the volume
group. I can't have udev activate that on sight, it will probably
corrupt data.

>>whoever wrote about editing the conf file?
>>i wrote about detecting that a device is already in user by
>>device-mapper and skipping that.
>
>Johnathan Brassow suggested adding the devices claimed by dmraid to 
>lvm's filter spec in its conf file.

probably because this is the only available way now.
anyway this method tends to suck when you have a complex setup.
and lvm device filters are not very intuitive to setup.

>What I am saying is to generalize your suggestion.  Rather than 
>specifically code the lvm tools to use the dm ioctls to check if a 
>device is in use and avoid using it, take a more general approach that 
>will work on similar problems as well, that don't involve device mapper 
>targeting the underlying device, and may be easier to implement.

Let's make a deal, you code that in udev, and in the meantime try to
convince Linus in ditching partition detection code from kernel.
I will code the filter for current lvm.

>If udev uses pvscan on each disk to find out if it is a member of a vg, 
>it can then note which vg it is a member of in its db.  Then it can 
>invoke lvm to attempt to activate that vg, explicitly telling lvm which 
>devices comprise the known pvs of that vg ( since it knows this 
>information ), rather than letting lvm scan /dev/sd*.  When dmraid 
>activates a raid set, udev can note that the physical disk is claimed by 
>dmraid, and it will never ask lvm to do anything with it.
doing this will require changing how lvm tools work.
1) disable the automatic vgscan performed by every lvm command
2) modify pvscan to be able to scan a single device

L.


-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Tue, Aug 07, 2007 at 11:17:47AM +0200, Luca Berra wrote:
> 1) disable the automatic vgscan performed by every lvm command

With access to a 100% accurate external device database instead of the
existing .cache this is easy.
The --trustcache argument would be enabled by default on such a system.

> 2) modify pvscan to be able to scan a single device

This would actually just be something like 'pvs -o uuid <device>' with
a subtle modification so it notices there are no VG fields required
and so it doesn't attempt to access the VG metadata (which might
require accessing disks other than the one specified).

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Fri, Aug 03, 2007 at 05:30:00PM -0400, Phillip Susi wrote:
> Editing a conf file to 
> specify a filter list of devices by name is all well and good for a 
> static system, but it does not play well in the modern udev managed plug 
> and play world.
 
BTW The filters could of course be generated dynamically (from a database) and
supplied on the command line (with --config) - or lvm2 could be extended to
read them from a database.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Fri, Aug 03, 2007 at 02:10:02PM -0400, Phillip Susi wrote:
> Where does lvm store state 
> information right now?  In conf files in /etc isn't it?  

It uses an untrusted cache in /etc/lvm as hints to speed up its
processing if the hints prove to be still correct.  It can't ever trust
because things can change on the system without lvm2's knowledge.  And
device scanning causes us no end of problems - particularly in a
clustered environment.  The new model should provide us with a
completely-reliable cache.

And why does lvm2 today have to include code to check for partition
tables, md devices, filesystems etc.?

If all these components co-operated through common interfaces there'd be
no need!

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: [dm-devel] Re: LVM on dmraid breakage

[Alasdair G Kergon]

On Thu, Aug 02, 2007 at 06:04:20PM -0400, Phillip Susi wrote:
> I'd rather not see it that tightly coupled to dm.  I was thinking 
> perhaps of something involving udev attributes so that udev would know 
> not to run pvscan on that device.  Though it looks like pvscan also 
> needs reworked so it plays nice with udev, being called to scan a given 
> device as detected rather than looking for all well known physical 
> device names in /dev.

It's been fairly clear how this should all work from the lvm2 side for
some time now, but it will still take a while to implement.

As I've said many times before, in my view the current problems stem
from some system components unilaterally switching to an event-driven
model while others (including lvm2 and initscripts) haven't - and
combining two incompatible models on a system was "brave".

If lvm2 is run in an event-driven environment, the idea is for it to
give up all its device scanning and filtering.  It will hand over
control of what devices to use to an external module shared by
everything that needs to know about devices - including mdadm,
initscripts, mount, hal, udev etc.  [What owns this module is irrelevant
but udev is an obvious candidate.]

When udev sees a new device, it will inform each subsystem of the
device.

lvm2 will provide an interface that is given a device and, if a PV is
present on it, lvm2 will pass information about it (including its UUID)
to the external module for storage.

The external module will have an interface capable of being told
by lvm2 (at any time) 'Devices with UUIDs X, Y and Z form a volume
group called VGn'.

Triggers can be defined so that when all the PV UUIDs are present
for a volume group with name VGn, an lvm2 command (like 'vgchange -ay
VGn') gets invoked.
 
You can see how md, mount etc. can work in a similar event-driven
fashion.

Alasdair
-- 
agk@redhat.com