[linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks?  Is this 
LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a 
symlink for the pool, but once I create a thin volume within the pool then the 
pool symlink disappears.

Thanks,
Chris

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Ilya Boka]

Command vgmknodes

On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen
<chris.friesen@windriver.com> wrote:
> Hi,
>
> Can someone tell me what creates the /dev/<volgroup>/<volume symlinks?  Is
> this LVM or udev (and if udev, do you know which rule)?
>
> I'm seeing some interesting behaviour where if I create a thin pool it
> creates a symlink for the pool, but once I create a thin volume within the
> pool then the pool symlink disappears.
>
> Thanks,
> Chris
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

If I run it I get this:

controller-1:/dev# vgmknodes -v
     Using logical volume(s) on command line.
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by 
udev but it was not found. Falling back to direct link creation.


So it thinks that udev should have already made it.  Also, the symlink it 
generates looks different than the symlinks already in /dev/cinder-volumes, in 
that it points to /dev/mapper/<name> rather than pointing to the "real" 
/dev/dm-X device like the others.


controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool -> 
/dev/mapper/cinder--volumes-cinder--volumes--pool
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
lrwxrwxrwx 1 root root  8 Jun 20 17:57 
volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18


Given the above, I think that something other than vgmknodes must be involved.

Chris


On 06/20/2016 03:03 PM, Ilya Boka wrote:
> Command vgmknodes
>
> On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen
> <chris.friesen@windriver.com> wrote:
>> Hi,
>>
>> Can someone tell me what creates the /dev/<volgroup>/<volume symlinks?  Is
>> this LVM or udev (and if udev, do you know which rule)?
>>
>> I'm seeing some interesting behaviour where if I create a thin pool it
>> creates a symlink for the pool, but once I create a thin volume within the
>> pool then the pool symlink disappears.
>>
>> Thanks,
>> Chris
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

Found it.  /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the 
/dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin 
volume in it it will exit without making a symlink because 
DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be 
nice if they agreed on whether or not a symlink should be created and if so 
where it should link to.

Chris


On 06/20/2016 03:53 PM, Chris Friesen wrote:
> If I run it I get this:
>
> controller-1:/dev# vgmknodes -v
>      Using logical volume(s) on command line.
>      Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
>      Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
>      Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
>      Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
>      Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
>    The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
> udev but it was not found. Falling back to direct link creation.
>
>
> So it thinks that udev should have already made it.  Also, the symlink it
> generates looks different than the symlinks already in /dev/cinder-volumes, in
> that it points to /dev/mapper/<name> rather than pointing to the "real"
> /dev/dm-X device like the others.
>
>
> controller-1:/dev# ls -l /dev/cinder-volumes/
> total 0
> lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
> lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool ->
> /dev/mapper/cinder--volumes-cinder--volumes--pool
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
> lrwxrwxrwx 1 root root  8 Jun 20 17:57
> volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18
>
>
> Given the above, I think that something other than vgmknodes must be involved.
>
> Chris
>
>
> On 06/20/2016 03:03 PM, Ilya Boka wrote:
>> Command vgmknodes
>>
>> On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen
>> <chris.friesen@windriver.com> wrote:
>>> Hi,
>>>
>>> Can someone tell me what creates the /dev/<volgroup>/<volume symlinks?  Is
>>> this LVM or udev (and if udev, do you know which rule)?
>>>
>>> I'm seeing some interesting behaviour where if I create a thin pool it
>>> creates a symlink for the pool, but once I create a thin volume within the
>>> pool then the pool symlink disappears.
>>>
>>> Thanks,
>>> Chris
>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

It appears that "vgscan --mknodes" also creates this symlink, and creates it 
pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.

controller-1:/dev# vgscan -v --mknodes
     Wiping cache of LVM-capable devices
     Wiping internal VG cache
   Reading all physical volumes.  This may take a while...
     Using volume group(s) on command line.
   Found volume group "nova-local" using metadata type lvm2
   Found volume group "cgts-vg" using metadata type lvm2
   Found volume group "cinder-volumes" using metadata type lvm2
     Using logical volume(s) on command line.
   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by 
udev but it was not found. Falling back to direct link creation.

controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool -> 
/dev/mapper/cinder--volumes-cinder--volumes--pool


This is a bad thing, because running the above or "vgmknodes" and then running 
"vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with 
a dangling symlink in it.

This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps 
erroneously) uses the existence of a non-empty /dev/<volume_group> directory as 
a test to see if the volume group is active or not.

Chris



On 06/20/2016 04:43 PM, Chris Friesen wrote:
> Found it.  /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the
> /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin
> volume in it it will exit without making a symlink because
> DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.
>
> Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be
> nice if they agreed on whether or not a symlink should be created and if so
> where it should link to.
>
> Chris
>
>
> On 06/20/2016 03:53 PM, Chris Friesen wrote:
>> If I run it I get this:
>>
>> controller-1:/dev# vgmknodes -v
>>      Using logical volume(s) on command line.
>>      Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
>>      Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
>>      Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
>>      Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
>>      Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
>>    The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
>> udev but it was not found. Falling back to direct link creation.
>>
>>
>> So it thinks that udev should have already made it.  Also, the symlink it
>> generates looks different than the symlinks already in /dev/cinder-volumes, in
>> that it points to /dev/mapper/<name> rather than pointing to the "real"
>> /dev/dm-X device like the others.
>>
>>
>> controller-1:/dev# ls -l /dev/cinder-volumes/
>> total 0
>> lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
>> lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool ->
>> /dev/mapper/cinder--volumes-cinder--volumes--pool
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
>> lrwxrwxrwx 1 root root  8 Jun 20 17:57
>> volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18
>>
>>
>> Given the above, I think that something other than vgmknodes must be involved.
>>
>> Chris
>>
>>
>> On 06/20/2016 03:03 PM, Ilya Boka wrote:
>>> Command vgmknodes
>>>
>>> On Mon, Jun 20, 2016 at 10:52 PM, Chris Friesen
>>> <chris.friesen@windriver.com> wrote:
>>>> Hi,
>>>>
>>>> Can someone tell me what creates the /dev/<volgroup>/<volume symlinks?  Is
>>>> this LVM or udev (and if udev, do you know which rule)?
>>>>
>>>> I'm seeing some interesting behaviour where if I create a thin pool it
>>>> creates a symlink for the pool, but once I create a thin volume within the
>>>> pool then the pool symlink disappears.
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> _______________________________________________
>>>> linux-lvm mailing list
>>>> linux-lvm@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/linux-lvm
>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>> _______________________________________________
>>> linux-lvm mailing list
>>> linux-lvm@redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-lvm
>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>>
>>
>

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):
> It appears that "vgscan --mknodes" also creates this symlink, and creates it
> pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.
>
> controller-1:/dev# vgscan -v --mknodes
>     Wiping cache of LVM-capable devices
>     Wiping internal VG cache
>   Reading all physical volumes.  This may take a while...
>     Using volume group(s) on command line.
>   Found volume group "nova-local" using metadata type lvm2
>   Found volume group "cgts-vg" using metadata type lvm2
>   Found volume group "cinder-volumes" using metadata type lvm2
>     Using logical volume(s) on command line.
>   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
> udev but it was not found. Falling back to direct link creation.
>
> controller-1:/dev# ls -l /dev/cinder-volumes/
> total 0
> lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
> lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
> /dev/mapper/cinder--volumes-cinder--volumes--pool
>
>
> This is a bad thing, because running the above or "vgmknodes" and then running
> "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory
> with a dangling symlink in it.
>
> This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps
> erroneously) uses the existence of a non-empty /dev/<volume_group> directory
> as a test to see if the volume group is active or not.
>


This whole thread is presentint  one continues sets of bugs and misunderstandings.


On modern Linux system - it should be ONLY udev to create any links in /dev 
dir. So  lvm2 does not create ANY links unless forced to do.

So the clear answer here is   'it is udev rule that creates links'


Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any 
duplicates.  If the same device (same UUID) is seen multiple times,
lvm2 has no way to know which of them is the right one to use.

Since you have not even shown the version of lvm2 in use and whather lvmetad 
is in use - there is no way to give you any more hints...

Regards

Zdenek

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

I'm using the stock CentOS7 version, I think.

   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
   Library version: 1.02.107-RHEL7 (2015-12-01)
   Driver version:  4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where 
udev is managing the symlinks?

Personally, I think that running that command should use the same logic as the 
udev rules to decide whether or not to create a symlink, and should create the 
symlink pointing to the same place as the udev rules.

I'm not sure what you're talking about as far as duplicates, I'm not seeing any 
duplicate devices.  The problem I see is that "vgscan --mknodes" or "vgmknodes" 
will both create an additional symlink when compared to the udev rules, and the 
additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|",  "a|/dev/sda6|", "r|.*|" ]

Chris


On 06/21/2016 03:07 AM, Zdenek Kabelac wrote:
> Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):
>> It appears that "vgscan --mknodes" also creates this symlink, and creates it
>> pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.
>>
>> controller-1:/dev# vgscan -v --mknodes
>>     Wiping cache of LVM-capable devices
>>     Wiping internal VG cache
>>   Reading all physical volumes.  This may take a while...
>>     Using volume group(s) on command line.
>>   Found volume group "nova-local" using metadata type lvm2
>>   Found volume group "cgts-vg" using metadata type lvm2
>>   Found volume group "cinder-volumes" using metadata type lvm2
>>     Using logical volume(s) on command line.
>>   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
>> udev but it was not found. Falling back to direct link creation.
>>
>> controller-1:/dev# ls -l /dev/cinder-volumes/
>> total 0
>> lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
>> lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
>> /dev/mapper/cinder--volumes-cinder--volumes--pool
>>
>>
>> This is a bad thing, because running the above or "vgmknodes" and then running
>> "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory
>> with a dangling symlink in it.
>>
>> This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps
>> erroneously) uses the existence of a non-empty /dev/<volume_group> directory
>> as a test to see if the volume group is active or not.
>>
>
>
> This whole thread is presentint  one continues sets of bugs and misunderstandings.
>
>
> On modern Linux system - it should be ONLY udev to create any links in /dev dir.
> So  lvm2 does not create ANY links unless forced to do.
>
> So the clear answer here is   'it is udev rule that creates links'
>
>
> Now the second part - you system is likely misconfigured.
> It's admin responsibility to filter out devices in a way lvm2 does not get any
> duplicates.  If the same device (same UUID) is seen multiple times,
> lvm2 has no way to know which of them is the right one to use.
>
> Since you have not even shown the version of lvm2 in use and whather lvmetad is
> in use - there is no way to give you any more hints...
>
> Regards
>
> Zdenek
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
> I'm using the stock CentOS7 version, I think.
>
>   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
>   Library version: 1.02.107-RHEL7 (2015-12-01)
>   Driver version:  4.33.0
>
> So are you saying that nobody should run "vgscan --mknodes" on a system where
> udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency',
udev doesn't work properly and you need links.

The links however will not be known to udev and likely whole system is
going to be crashing soon or is misconfigured in major way.


> I'm not sure what you're talking about as far as duplicates, I'm not seeing
> any duplicate devices.  The problem I see is that "vgscan --mknodes" or
> "vgmknodes" will both create an additional symlink when compared to the udev
> rules, and the additional symlink is not deleted when I deactivate the volume
> group.
>
> For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and
>
> global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|",  "a|/dev/sda6|", "r|.*|" ]

And now we are getting to the point on your problem:

"a|/dev/sda|"   will also match "a|/dev/sda6|"
(and /dev/sda5...)

If you would like to get only 'exact' '/dev/sda' you would need to use

^/dev/sda$

otherwise '/dev/sda' may appear anywhere as substring of your device path.

Regards

Zdenek

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

On 06/22/2016 03:23 AM, Zdenek Kabelac wrote:
> Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
>> I'm using the stock CentOS7 version, I think.
>>
>>   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
>>   Library version: 1.02.107-RHEL7 (2015-12-01)
>>   Driver version:  4.33.0
>>
>> So are you saying that nobody should run "vgscan --mknodes" on a system where
>> udev is managing the symlinks?
>
> Yes - on such system this command should be used only in case of 'emergency',
> udev doesn't work properly and you need links.
>
> The links however will not be known to udev and likely whole system is
> going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed.  But even so wouldn't it 
make sense to have vgscan use the same logic as udev in terms of what symlinks 
to make and where to make them to?


>> I'm not sure what you're talking about as far as duplicates, I'm not seeing
>> any duplicate devices.  The problem I see is that "vgscan --mknodes" or
>> "vgmknodes" will both create an additional symlink when compared to the udev
>> rules, and the additional symlink is not deleted when I deactivate the volume
>> group.
>>
>> For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and
>>
>> global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|",  "a|/dev/sda6|", "r|.*|" ]
>
> And now we are getting to the point on your problem:
>
> "a|/dev/sda|"   will also match "a|/dev/sda6|"
> (and /dev/sda5...)
>
> If you would like to get only 'exact' '/dev/sda' you would need to use
>
> ^/dev/sda$
>
> otherwise '/dev/sda' may appear anywhere as substring of your device path.


Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices that 
form part of various separate volume groups.  Really it's the /dev/sda6 that is 
extra here since it's already matched.

The problematic issue I'm seeing with udev/vgscan is actually with the volume 
group on /dev/drbd4.

I really don't think I'm having any problems with duplicates...just with vgscan 
creating extra symlinks compared to udev, that udev doesn't clean up.

Chris

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
> On 06/22/2016 03:23 AM, Zdenek Kabelac wrote:
>> Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
>>> I'm using the stock CentOS7 version, I think.
>>>
>>>   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
>>>   Library version: 1.02.107-RHEL7 (2015-12-01)
>>>   Driver version:  4.33.0
>>>
>>> So are you saying that nobody should run "vgscan --mknodes" on a system where
>>> udev is managing the symlinks?
>>
>> Yes - on such system this command should be used only in case of 'emergency',
>> udev doesn't work properly and you need links.
>>
>> The links however will not be known to udev and likely whole system is
>> going to be crashing soon or is misconfigured in major way.
>
> Okay, I'll see if I can get the call to vgscan removed.  But even so wouldn't
> it make sense to have vgscan use the same logic as udev in terms of what
> symlinks to make and where to make them to?

It *IS* using same logic.

If the link is not there - the bug is in your udev rules.

When udev is properly configured, vgscan should not show missing link.

It's worth to note -  'dm' subsystem is in general 'ignored' by udev and 'dm' 
user has to provide proper rules.  If your drbd device is missing,
then something wrong is with rules...

Likely 'drbd' devices are not properly scanned by udev (and I assume they 
would not even work on system with  use_lvmetad=1).


>>> I'm not sure what you're talking about as far as duplicates, I'm not seeing
>>> any duplicate devices.  The problem I see is that "vgscan --mknodes" or
>>> "vgmknodes" will both create an additional symlink when compared to the udev
>>> rules, and the additional symlink is not deleted when I deactivate the volume
>>> group.
>>>
>>> For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and
>>>
>>> global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|",  "a|/dev/sda6|", "r|.*|" ]
>>
>> And now we are getting to the point on your problem:
>>
>> "a|/dev/sda|"   will also match "a|/dev/sda6|"
>> (and /dev/sda5...)
>>
>> If you would like to get only 'exact' '/dev/sda' you would need to use
>>
>> ^/dev/sda$
>>
>> otherwise '/dev/sda' may appear anywhere as substring of your device path.
>
>
> Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices
> that form part of various separate volume groups.  Really it's the /dev/sda6
> that is extra here since it's already matched.
>
> The problematic issue I'm seeing with udev/vgscan is actually with the volume
> group on /dev/drbd4.
>
> I really don't think I'm having any problems with duplicates...just with
> vgscan creating extra symlinks compared to udev, that udev doesn't clean up.
>

Ahh - ok then - I thought you wanted to see only sda6 on your machine.


Regards

Zdenek

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

On 06/23/2016 02:34 AM, Zdenek Kabelac wrote:
> Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
>> On 06/22/2016 03:23 AM, Zdenek Kabelac wrote:
>>> Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
>>>> I'm using the stock CentOS7 version, I think.
>>>>
>>>>   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
>>>>   Library version: 1.02.107-RHEL7 (2015-12-01)
>>>>   Driver version:  4.33.0
>>>>
>>>> So are you saying that nobody should run "vgscan --mknodes" on a system where
>>>> udev is managing the symlinks?
>>>
>>> Yes - on such system this command should be used only in case of 'emergency',
>>> udev doesn't work properly and you need links.
>>>
>>> The links however will not be known to udev and likely whole system is
>>> going to be crashing soon or is misconfigured in major way.
>>
>> Okay, I'll see if I can get the call to vgscan removed.  But even so wouldn't
>> it make sense to have vgscan use the same logic as udev in terms of what
>> symlinks to make and where to make them to?
>
> It *IS* using same logic.
>
> If the link is not there - the bug is in your udev rules.
>
> When udev is properly configured, vgscan should not show missing link.

It doesn't seem to work this way in practice on a stock CentOS system.  Here's 
the sequence:

1) create a volume group:
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.

2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"

Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9

3) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"

Now the link for the thin pool itself has disappeared:
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to 
dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)


4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing 
to the /dev/mapper name instead of directly to the /dev/dm-*.  Also, it's 
indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:

[root@centos7 centos]# vgscan --mknodes
   Configuration setting "snapshot_autoextend_percent" invalid. It's not part of 
any section.
   Configuration setting "snapshot_autoextend_threshold" invalid. It's not part 
of any section.
   Reading all physical volumes.  This may take a while...
   Found volume group "chris-volumes" using metadata type lvm2
   Found volume group "centos" using metadata type lvm2
   Found volume group "cinder-volumes" using metadata type lvm2
   The link /dev/chris-volumes/chris-volumes-pool should have been created by 
udev but it was not found. Falling back to direct link creation.
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> 
/dev/mapper/chris--volumes-chris--volumes--pool
lrwxrwxrwx. 1 root root  8 Jun 23 12:23 thinvolume -> ../dm-11


5) If I run "vgchange -an chris-volumes", I'm left with a /dev/chris-volumes 
directory containing a broken symlink (broken because 
/dev/mapper/chris--volumes-chris--volumes--pool doesn't exist anymore):

[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> 
/dev/mapper/chris--volumes-chris--volumes--pool



It looks like /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the 
/dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin 
volume in it it will exit without making a symlink because 
DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

[root@centos7 centos]# udevadm info /dev/chris-volumes/chris-volumes-pool
P: /devices/virtual/block/dm-10
N: dm-10
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1


Chris

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
> On 06/23/2016 02:34 AM, Zdenek Kabelac wrote:
>> Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
>>> On 06/22/2016 03:23 AM, Zdenek Kabelac wrote:
>>>> Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
>>>>> I'm using the stock CentOS7 version, I think.
>>>>>
>>>>>   LVM version:     2.02.130(2)-RHEL7 (2015-12-01)
>>>>>   Library version: 1.02.107-RHEL7 (2015-12-01)
>>>>>   Driver version:  4.33.0
>>>>>
>>>>> So are you saying that nobody should run "vgscan --mknodes" on a system
>>>>> where
>>>>> udev is managing the symlinks?
>>>>
>>>> Yes - on such system this command should be used only in case of 'emergency',
>>>> udev doesn't work properly and you need links.
>>>>
>>>> The links however will not be known to udev and likely whole system is
>>>> going to be crashing soon or is misconfigured in major way.
>>>
>>> Okay, I'll see if I can get the call to vgscan removed.  But even so wouldn't
>>> it make sense to have vgscan use the same logic as udev in terms of what
>>> symlinks to make and where to make them to?
>>
>> It *IS* using same logic.
>>
>> If the link is not there - the bug is in your udev rules.
>>
>> When udev is properly configured, vgscan should not show missing link.
>
> It doesn't seem to work this way in practice on a stock CentOS system.  Here's
> the sequence:
>
> 1) create a volume group:
> "vgcreate chris-volumes /dev/loop2"
> At this point there is no /dev/chris-volumes directory.
>

ONLY  active volume do have links

Never ever there is supposed to be directory entry for VG without any active LV.

There is not a term 'active VG' as such - it always is in connection with 
active LV - thus directory without any active LV inside is pure bug - if you 
see it - report it as regular BZ..



> 2) Create a thin pool in the volume group:
> "lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"
>
> Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
> [root@centos7 centos]# ls -l /dev/chris-volumes
> total 0
> lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9
>
> 3) Create a thin volume in the thin pool:
> "lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"
>
> Now the link for the thin pool itself has disappeared:
> [root@centos7 centos]# ls -l /dev/chris-volumes
> total 0
> lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11
>
> (At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to
> dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)

correct

> 4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing
> to the /dev/mapper name instead of directly to the /dev/dm-*.  Also, it's
> indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:

Only 'final' resolving device does matter.

if link is     /dev/vg/lv  ->   /dev/mapper/vg-lv   ->  /dev/dm-XXX
or   /dev/vg/lv  ->  /dev/dm-XXX  - it does not matter.

There are some 'tricks' related to thin-pool maintainance.

Unused  thin-pool is 'public' LV - has  /dev/vg/lv   link.
Used thin-pool by lvm2 is 'private'  LV -   doesn't have  /dev/vg/lv  link.

All device should always have entry in  /dev/mapper/ - either links
to real devices or direct nodes (on older systems)

lvm2 users are always supposed to use ONLY /dev/vg/lv  devices for access.




>
> [root@centos7 centos]# vgscan --mknodes
>   Configuration setting "snapshot_autoextend_percent" invalid. It's not part
> of any section.
>   Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
> of any section.

fix your lvm.conf   (uncomment sections)

>   Reading all physical volumes.  This may take a while...
>   Found volume group "chris-volumes" using metadata type lvm2
>   Found volume group "centos" using metadata type lvm2
>   Found volume group "cinder-volumes" using metadata type lvm2
>   The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints
link creation for this case.

There should not have been  /dev/vg/pool link - this is correctly marked
for udev - but incorrectly for udev validation.

However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without 
this extra device in table.


Regards

Zdenek

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Chris Friesen]

On 06/23/2016 11:21 AM, Zdenek Kabelac wrote:
> Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):

>> [root@centos7 centos]# vgscan --mknodes
>>   Configuration setting "snapshot_autoextend_percent" invalid. It's not part
>> of any section.
>>   Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
>> of any section.
>
> fix your lvm.conf   (uncomment sections)
>
>>   Reading all physical volumes.  This may take a while...
>>   Found volume group "chris-volumes" using metadata type lvm2
>>   Found volume group "centos" using metadata type lvm2
>>   Found volume group "cinder-volumes" using metadata type lvm2
>>   The link /dev/chris-volumes/chris-volumes-pool should have been created by
>
> Ok - there seems to be internal bug in lvm2 - which incorrectly hints
> link creation for this case.
>
> There should not have been  /dev/vg/pool link - this is correctly marked
> for udev - but incorrectly for udev validation.
>
> However the bug is actually not so much important - it just links
> to 'wrapper' device - and eventually we will resolve the problem even without
> this extra device in table.

The problem that it causes for me is that when I run "vgchange -an 
chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it 
because udev doesn't remove the symlink added by vgscan.

This causes the LVM OCF script in the "resource-agents" package to break, 
because it is using the existance of the /dev/vg directory as a proxy for 
whether the volume group is active (or really as you said earlier, whether there 
are active volumes within the volume group).

I reported this as a bug to the "resource-agents" package developers, and they 
said that they can't actually call lvm commands in their "status" routines 
because there have been cases where clustered LVM hung when querying status, 
causing the OCF script to hang and monitoring to fail.

Ultimately I'll see if I can work around it by not calling "vgscan --mknodes". 
Originally it was added in to fix some problems, but that was a while back so 
things may behave properly now.

Thanks for your help,
Chris

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):
> On 06/23/2016 11:21 AM, Zdenek Kabelac wrote:
>> Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
>
>>> [root@centos7 centos]# vgscan --mknodes
>>>   Configuration setting "snapshot_autoextend_percent" invalid. It's not part
>>> of any section.
>>>   Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
>>> of any section.
>>
>> fix your lvm.conf   (uncomment sections)
>>
>>>   Reading all physical volumes.  This may take a while...
>>>   Found volume group "chris-volumes" using metadata type lvm2
>>>   Found volume group "centos" using metadata type lvm2
>>>   Found volume group "cinder-volumes" using metadata type lvm2
>>>   The link /dev/chris-volumes/chris-volumes-pool should have been created by
>>
>> Ok - there seems to be internal bug in lvm2 - which incorrectly hints
>> link creation for this case.
>>
>> There should not have been  /dev/vg/pool link - this is correctly marked
>> for udev - but incorrectly for udev validation.
>>
>> However the bug is actually not so much important - it just links
>> to 'wrapper' device - and eventually we will resolve the problem even without
>> this extra device in table.
>
> The problem that it causes for me is that when I run "vgchange -an
> chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it
> because udev doesn't remove the symlink added by vgscan.

Yep - as said - in normal circumstance  use should NOT run 'vgmknodes'
as the created links will not be known/visible to udev  - so this behavior is 
correct.

> This causes the LVM OCF script in the "resource-agents" package to break,
> because it is using the existance of the /dev/vg directory as a proxy for
> whether the volume group is active (or really as you said earlier, whether
> there are active volumes within the volume group).
>
> I reported this as a bug to the "resource-agents" package developers, and they
> said that they can't actually call lvm commands in their "status" routines
> because there have been cases where clustered LVM hung when querying status,
> causing the OCF script to hang and monitoring to fail.
>
> Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".

yes please,  start with this one...

vgmknodes are really supposed to be used on some unique urgent problem - not 
executed in script every hour...

But yes - lvm2 will need to fix link creation of pool-in-use...

> Originally it was added in to fix some problems, but that was a while back so
> things may behave properly now.

Regards

Zdenek

Re: [linux-lvm] what creates the symlinks in /dev/<volgroup> ?

[Zdenek Kabelac]

Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):
> On 06/23/2016 11:21 AM, Zdenek Kabelac wrote:
>> Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
>
>>> [root@centos7 centos]# vgscan --mknodes
>>>   Configuration setting "snapshot_autoextend_percent" invalid. It's not part
>>> of any section.
>>>   Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
>>> of any section.
>>
>> fix your lvm.conf   (uncomment sections)
>>
>>>   Reading all physical volumes.  This may take a while...
>>>   Found volume group "chris-volumes" using metadata type lvm2
>>>   Found volume group "centos" using metadata type lvm2
>>>   Found volume group "cinder-volumes" using metadata type lvm2
>>>   The link /dev/chris-volumes/chris-volumes-pool should have been created by
>>
>> Ok - there seems to be internal bug in lvm2 - which incorrectly hints
>> link creation for this case.
>>
>> There should not have been  /dev/vg/pool link - this is correctly marked
>> for udev - but incorrectly for udev validation.
>>
>> However the bug is actually not so much important - it just links
>> to 'wrapper' device - and eventually we will resolve the problem even without
>> this extra device in table.
>
> The problem that it causes for me is that when I run "vgchange -an
> chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it
> because udev doesn't remove the symlink added by vgscan.
>
> This causes the LVM OCF script in the "resource-agents" package to break,
> because it is using the existance of the /dev/vg directory as a proxy for
> whether the volume group is active (or really as you said earlier, whether
> there are active volumes within the volume group).
>
> I reported this as a bug to the "resource-agents" package developers, and they
> said that they can't actually call lvm commands in their "status" routines
> because there have been cases where clustered LVM hung when querying status,
> causing the OCF script to hang and monitoring to fail.
>
> Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".
> Originally it was added in to fix some problems, but that was a while back so
> things may behave properly now.

Should be resolved with this upstream commit:

https://www.redhat.com/archives/lvm-devel/2016-June/msg00187.html

Regards

Zdenek