[PATCH] config: set external_device_info_source=none if udev isn't running

[PATCH] config: set external_device_info_source=none if udev isn't running

[mwilck]

From: Martin Wilck <mwilck@suse.com>

LVM2 has several configuration options related to device detection
and udev. In particular, we have obtain_device_list_from_udev=(0|1)
and external_device_info_source=("none"|"udev"). The two options are
obviously semantically related, but it's rather unclear if and how
they interact.

If udev is unavailable, e.g. in containers, obtain_device_list_from_udev
(which defaults to 1) will be automatically reset to 0. However,
if external_device_info_source="udev" is set, this setting is not
reset to "none", leading to error messages like

  Udev database has incomplete information about device /dev/vda.
  /dev/vda: Failed to get external handle [udev].

This patch changes that, treating external_device_info_source the
same way as obtain_device_list_from_udev, thereby making LVM2's
device detection more consistent.

The default for external_device_info_source is "none", but I believe
there are very good reasons to change this setting to "udev", because
LVM will get detection of multipath and md devices wrong most of the
time otherwise. LVM should follow the same logic as systemd and other
system components with respect to device detection.

Signed-off-by: Martin Wilck <mwilck@suse.com>
---
 lib/commands/toolcontext.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/lib/commands/toolcontext.c b/lib/commands/toolcontext.c
index 63b6811..55f6806 100644
--- a/lib/commands/toolcontext.c
+++ b/lib/commands/toolcontext.c
@@ -598,9 +598,20 @@ static int _process_config(struct cmd_context *cmd)
 	dev_ext_info_src = find_config_tree_str(cmd, devices_external_device_info_source_CFG, NULL);
 	if (dev_ext_info_src && !strcmp(dev_ext_info_src, "none"))
 		init_external_device_info_source(DEV_EXT_NONE);
-	else if (dev_ext_info_src && !strcmp(dev_ext_info_src, "udev"))
-		init_external_device_info_source(DEV_EXT_UDEV);
-	else {
+	else if (dev_ext_info_src && !strcmp(dev_ext_info_src, "udev")) {
+		/*
+		 * Override existing config and hardcode external_device_info_source=none if:
+		 *   - udev is not running
+		 *   - udev is disabled using DM_DISABLE_UDEV environment variable
+		 * See also treatment of obtain_device_list_from_udev in _init_dev_cache()
+		 */
+		if (udev_is_running())
+			init_external_device_info_source(DEV_EXT_UDEV);
+		else  {
+			log_notice("udev is not running. Using external_device_info_source=none");
+			init_external_device_info_source(DEV_EXT_NONE);
+		}
+	} else {
 		log_error("Invalid external device info source specification.");
 		return 0;
 	}
-- 
2.29.2

[PATCH] config: set external_device_info_source=none if udev isn't running

[Zdenek Kabelac]

Dne 27. 01. 21 v 18:28 mwilck at suse.com napsal(a):
> From: Martin Wilck <mwilck@suse.com>
> 
> LVM2 has several configuration options related to device detection
> and udev. In particular, we have obtain_device_list_from_udev=(0|1)
> and external_device_info_source=("none"|"udev"). The two options are
> obviously semantically related, but it's rather unclear if and how
> they interact.
> 
> If udev is unavailable, e.g. in containers, obtain_device_list_from_udev
> (which defaults to 1) will be automatically reset to 0. However,
> if external_device_info_source="udev" is set, this setting is not
> reset to "none", leading to error messages like
> 
>    Udev database has incomplete information about device /dev/vda.
>    /dev/vda: Failed to get external handle [udev].
> 
> This patch changes that, treating external_device_info_source the
> same way as obtain_device_list_from_udev, thereby making LVM2's
> device detection more consistent.
> 
> The default for external_device_info_source is "none", but I believe
> there are very good reasons to change this setting to "udev", because
> LVM will get detection of multipath and md devices wrong most of the
> time otherwise. LVM should follow the same logic as systemd and other


Hi

I'm afraid there is no such simple fix for this as you might think.

In general lvm2 should NOT be used in containers - containers are NOT
technology for such use-case - for as long as devices are not 'a 
containerized' resources.

There have been already several attempts with usage of 'privileged' container,
but still the is 'generic' problem about synchronization between the
main system and its containers.

If users need devices in their containers - those should be always created
on hosting machine with proper naming and access control and given to the 
container. It's fairly bad plan to give such 'power' to the container as it 
can very simply make the host machine (with all other containers) unusable.

But if there is anyone using 'lvm2' inside a container today he must always 
use non-udev variant of lvm2 with some 'strict' rules all over the hosting
machine and other containers - but I'd highly discourage such usage as it
requires a very good 'insight' knowledge.

Zdenek

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Thu, 2021-01-28 at 11:10 +0100, Zdenek Kabelac wrote:
> Dne 27. 01. 21 v 18:28 mwilck at suse.com?napsal(a):
> > From: Martin Wilck <mwilck@suse.com>
> > 
> > LVM2 has several configuration options related to device detection
> > and udev. In particular, we have obtain_device_list_from_udev=(0|1)
> > and external_device_info_source=("none"|"udev"). The two options
> > are
> > obviously semantically related, but it's rather unclear if and how
> > they interact.
> > 
> > If udev is unavailable, e.g. in containers,
> > obtain_device_list_from_udev
> > (which defaults to 1) will be automatically reset to 0. However,
> > if external_device_info_source="udev" is set, this setting is not
> > reset to "none", leading to error messages like
> > 
> > ?? Udev database has incomplete information about device /dev/vda.
> > ?? /dev/vda: Failed to get external handle [udev].
> > 
> > This patch changes that, treating external_device_info_source the
> > same way as obtain_device_list_from_udev, thereby making LVM2's
> > device detection more consistent.
> > 
> > The default for external_device_info_source is "none", but I
> > believe
> > there are very good reasons to change this setting to "udev",
> > because
> > LVM will get detection of multipath and md devices wrong most of
> > the
> > time otherwise. LVM should follow the same logic as systemd and
> > other
> 
> 
> Hi
> 
> I'm afraid there is no such simple fix for this as you might think.
> 
> In general lvm2 should NOT be used in containers - containers are NOT
> technology for such use-case - for as long as devices are not 'a 
> containerized' resources.
> 
> There have been already several attempts with usage of 'privileged'
> container,
> but still the is 'generic' problem about synchronization between the
> main system and its containers.
> 
> If users need devices in their containers - those should be always
> created
> on hosting machine with proper naming and access control and given to
> the 
> container. It's fairly bad plan to give such 'power' to the container
> as it 
> can very simply make the host machine (with all other containers)
> unusable.
> 
> But if there is anyone using 'lvm2' inside a container today he must
> always 
> use non-udev variant of lvm2 with some 'strict' rules all over the
> hosting
> machine and other containers - but I'd highly discourage such usage
> as it
> requires a very good 'insight' knowledge.

I fully agree. 

But does that mean my patch is wrong? Don't you agree that the
different handling of obtain_device_list_from_udev and
external_device_info_source in the current code is inconsistent?


IMO this should actually not be a matter of configuration. Fallback
should be used if an only if udev is unavailable. The only reason not
to do it that way is backward compatibility. Or am I missing something?

The background of this patch is not that I want to enable use of LVM in
containers. It is that we (openSUSE/SUSE) have recently made
external_device_info_source="udev" the default. We have done this
because we have repeatedly found that multipath and MD component
detection with LVM was unreliable with "none". That't not surprising,
because if "none" is used, LVM simply doesn't use the same logic as the
rest of the system, in particular systemd.

The problem is now that there are some areas where LVM is being used
without udev, and we're getting bug reports from those users. Examples
are chroot environments for building images. So, we need a way to
reconcile the (IMO nonetheless reasonable) changed default of
external_device_info_source="udev" with the requirements of these
special environments. The users of these environments were obviously
using LVM in ways that worked without having udev fully available,
otherwise their stuff hadn't ever worked; so your general concerns
don't apply here.

This sounds like I'm looking for a "hack" as workaround for a mistake
we made, and it's partially true.

Still, I think the patch is not a hack, but generally correct, and has
the pleasant side effect to fix our issue.

Regards
Martin

[PATCH] config: set external_device_info_source=none if udev isn't running

[Zdenek Kabelac]

Dne 28. 01. 21 v 11:27 Martin Wilck napsal(a):
> On Thu, 2021-01-28 at 11:10 +0100, Zdenek Kabelac wrote:
>> Dne 27. 01. 21 v 18:28 mwilck at suse.com?napsal(a):
>>> From: Martin Wilck <mwilck@suse.com>
>>>
>>> LVM2 has several configuration options related to device detection
>>> and udev. In particular, we have obtain_device_list_from_udev=(0|1)
>>> and external_device_info_source=("none"|"udev"). The two options
>>> are
>>> obviously semantically related, but it's rather unclear if and how
>>> they interact.
>>>
>>> If udev is unavailable, e.g. in containers,
>>> obtain_device_list_from_udev
>>> (which defaults to 1) will be automatically reset to 0. However,
>>> if external_device_info_source="udev" is set, this setting is not
>>> reset to "none", leading to error messages like
>>>
>>>  ?? Udev database has incomplete information about device /dev/vda.
>>>  ?? /dev/vda: Failed to get external handle [udev].
>>>
>>> This patch changes that, treating external_device_info_source the
>>> same way as obtain_device_list_from_udev, thereby making LVM2's
>>> device detection more consistent.
>>>
>>> The default for external_device_info_source is "none", but I
>>> believe
>>> there are very good reasons to change this setting to "udev",
>>> because
>>> LVM will get detection of multipath and md devices wrong most of
>>> the
>>> time otherwise. LVM should follow the same logic as systemd and
>>> other
>>
>>
>> Hi
>>
>> I'm afraid there is no such simple fix for this as you might think.
>>
>> 
> But does that mean my patch is wrong? Don't you agree that the
> different handling of obtain_device_list_from_udev and
> external_device_info_source in the current code is inconsistent?
>

Hi

One of the main point probably is  -

if udev is not working on your main system - and should - you should
get it working first.

Side case can be - you run lvm2 command - and someone 'restarts' udev
(i.e. via upgrade...)

So in general - this fallback should be only like new configurable
option - since normally you do not want lvm2 to ever touch /dev
dir which is under udev control.

Adding your 'fallback' adds some level of randomness and diggers
possibly bigger hole of troubles in system.

> The background of this patch is not that I want to enable use of LVM in
> containers. It is that we (openSUSE/SUSE) have recently made

Yep - everyone thinks running lvm2 in Container is 'smart' :)

> Still, I think the patch is not a hack, but generally correct, and has
> the pleasant side effect to fix our issue.

It would need some 'new' mode - but IMHO that's then equivalent
to setting correct mode directly.

What can likely work better is to add some 'detection' of being executed
in container -  scream at user and do maybe that 'udev' hack fallback ;)

Regards

Zdenek

[PATCH] config: set external_device_info_source=none if udev isn't running

[heming.zhao]

On 1/29/21 6:56 AM, Zdenek Kabelac wrote:
> Dne 28. 01. 21 v 11:27 Martin Wilck napsal(a):
>> On Thu, 2021-01-28 at 11:10 +0100, Zdenek Kabelac wrote:
>>> Dne 27. 01. 21 v 18:28 mwilck at suse.com?napsal(a):
>>>> From: Martin Wilck <mwilck@suse.com>
>>>>
>>>> LVM2 has several configuration options related to device detection
>>>> and udev. In particular, we have obtain_device_list_from_udev=(0|1)
>>>> and external_device_info_source=("none"|"udev"). The two options
>>>> are
>>>> obviously semantically related, but it's rather unclear if and how
>>>> they interact.
>>>>
>>>> If udev is unavailable, e.g. in containers,
>>>> obtain_device_list_from_udev
>>>> (which defaults to 1) will be automatically reset to 0. However,
>>>> if external_device_info_source="udev" is set, this setting is not
>>>> reset to "none", leading to error messages like
>>>>
>>>> ??? Udev database has incomplete information about device /dev/vda.
>>>> ??? /dev/vda: Failed to get external handle [udev].
>>>>
>>>> This patch changes that, treating external_device_info_source the
>>>> same way as obtain_device_list_from_udev, thereby making LVM2's
>>>> device detection more consistent.
>>>>
>>>> The default for external_device_info_source is "none", but I
>>>> believe
>>>> there are very good reasons to change this setting to "udev",
>>>> because
>>>> LVM will get detection of multipath and md devices wrong most of
>>>> the
>>>> time otherwise. LVM should follow the same logic as systemd and
>>>> other
>>>
>>>
>>> Hi
>>>
>>> I'm afraid there is no such simple fix for this as you might think.
>>>
>>>
>> But does that mean my patch is wrong? Don't you agree that the
>> different handling of obtain_device_list_from_udev and
>> external_device_info_source in the current code is inconsistent?
>>
> 
> Hi
> 
> One of the main point probably is? -
> 
> if udev is not working on your main system - and should - you should
> get it working first.
> 
> Side case can be - you run lvm2 command - and someone 'restarts' udev
> (i.e. via upgrade...)
> 
> So in general - this fallback should be only like new configurable
> option - since normally you do not want lvm2 to ever touch /dev
> dir which is under udev control.
> 
> Adding your 'fallback' adds some level of randomness and diggers
> possibly bigger hole of troubles in system.
> 

 From above logic, lvm2 should set 'external_device_info_source = "udev"' as default,
but real world is not.
why? I guess it's for compatible with old systems which can't run udev.
Whether or not we would change this item from "none" to "udev" in main branch?

>> The background of this patch is not that I want to enable use of LVM in
>> containers. It is that we (openSUSE/SUSE) have recently made
> 
> Yep - everyone thinks running lvm2 in Container is 'smart' :)
> 
>> Still, I think the patch is not a hack, but generally correct, and has
>> the pleasant side effect to fix our issue.
> 
> It would need some 'new' mode - but IMHO that's then equivalent
> to setting correct mode directly.
> 
> What can likely work better is to add some 'detection' of being executed
> in container -? scream at user and do maybe that 'udev' hack fallback ;)
> 

Regards
Heming

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

Hello Zdenek,

On Thu, 2021-01-28 at 23:56 +0100, Zdenek Kabelac wrote:
> Dne 28. 01. 21 v 11:27 Martin Wilck napsal(a):
> > On Thu, 2021-01-28 at 11:10 +0100, Zdenek Kabelac wrote:
> > > Dne 27. 01. 21 v 18:28 mwilck at suse.com?napsal(a):
> > > > From: Martin Wilck <mwilck@suse.com>
> > > > 
> > > > LVM2 has several configuration options related to device
> > > > detection
> > > > and udev. In particular, we have
> > > > obtain_device_list_from_udev=(0|1)
> > > > and external_device_info_source=("none"|"udev"). The two
> > > > options
> > > > are
> > > > obviously semantically related, but it's rather unclear if and
> > > > how
> > > > they interact.
> > > > 
> > > > If udev is unavailable, e.g. in containers,
> > > > obtain_device_list_from_udev
> > > > (which defaults to 1) will be automatically reset to 0.
> > > > However,
> > > > if external_device_info_source="udev" is set, this setting is
> > > > not
> > > > reset to "none", leading to error messages like
> > > > 
> > > > ??? Udev database has incomplete information about device
> > > > /dev/vda.
> > > > ??? /dev/vda: Failed to get external handle [udev].
> > > > 
> > > > This patch changes that, treating external_device_info_source
> > > > the
> > > > same way as obtain_device_list_from_udev, thereby making LVM2's
> > > > device detection more consistent.
> > > > 
> > > > The default for external_device_info_source is "none", but I
> > > > believe
> > > > there are very good reasons to change this setting to "udev",
> > > > because
> > > > LVM will get detection of multipath and md devices wrong most
> > > > of
> > > > the
> > > > time otherwise. LVM should follow the same logic as systemd and
> > > > other
> > > 
> > > 
> > > Hi
> > > 
> > > I'm afraid there is no such simple fix for this as you might
> > > think.
> > > 
> > > 
> > But does that mean my patch is wrong? Don't you agree that the
> > different handling of obtain_device_list_from_udev and
> > external_device_info_source in the current code is inconsistent?
> > 
> 
> Hi
> 
> One of the main point probably is? -
> 
> if udev is not working on your main system - and should - you should
> get it working first.
> 

Of course *udev* works "in my main system". But *LVM2* does not: with
the default setting "external_device_info_source=none", it ignores udev
properties of devices. This is the source of lots of subtle errors and
race conditions during device setup. Therefore we changed the setting
to "udev".

How do you handle that in Fedora? I took the liberty to look at the
Fedora 33 package, and it doesn't change default from "none" to
"udev".?So by common sense, Fedora is going to suffer from the same
general problem that (open)SUSE sees: With "none", lvm can detect
multipath or MD components only "after the fact", i.e. after multipathd
or mdadm have grabbed them already. If pvscan and multipathd start up
simultaneously, it's anyones guess who "wins" (*). With "udev", that
can't happen, and that's why "udev" should be made the default.

(I'm cc'ing Ben Marzinski, as he should know this problem very well,
and knows Fedora, too).

> Side case can be - you run lvm2 command - and someone 'restarts' udev
> (i.e. via upgrade...)

That shouldn't a problem AFAICS, because libudev only looks at the udev
data base, which is unaffected by updates.

> So in general - this fallback should be only like new configurable
> option - since normally you do not want lvm2 to ever touch /dev
> dir which is under udev control.

IIUC you argue for disallowing the fallback by default.?
I'm ok with that.

But that would also mean that you would have to change the default to
"udev", and *remove* both options "external_device_info_source" and
"obtain_device_list_from_udev". The former should be hard coded to
"udev" and the latter to 1, end of story. If you don't remove these
options, how would the new option interact with the existing two? Which
would take precedence?

> Adding your 'fallback' adds some level of randomness and diggers
> possibly bigger hole of troubles in system.

Explain that, please. The fallback does nothing in the current default
case (external_device_info_source="none"). And in the "udev" case, it
avoids an error condition in special situations, simply by falling back
to the current default. What's wrong about that?

> > Still, I think the patch is not a hack, but generally correct, and
> > has
> > the pleasant side effect to fix our issue.
> 
> It would need some 'new' mode - but IMHO that's then equivalent
> to setting correct mode directly.
> 
> What can likely work better is to add some 'detection' of being
> executed
> in container -? scream at user and do maybe that 'udev' hack fallback
> ;)

Again, this is not only about containers, but any environment where 
the udev data base is not available.

If you can provide a better solution than my patch, we'll happily
take it. But we need *something* to fix the current breakage.

Best regards,
Martin



(*) The problem is mitigated on modern distros by the fact that pvscan
is started by systemd, which of course honors SYSTEMD_READY. This is
the reason actual problems are encountered rarely, and only on sytems
with complex storage setup and lots of LUNs and PVs. But races can
occur, because pvscan, once started, tries to build up a device data
base, and while doing that, it relies on its own device detection
scheme, which is inconsistent with systemd's unless
"external_device_info_source" is set to "udev".

[PATCH] config: set external_device_info_source=none if udev isn't running

[Zdenek Kabelac]

Dne 29. 01. 21 v 11:07 Martin Wilck napsal(a):
> Hello Zdenek,
> 
> On Thu, 2021-01-28 at 23:56 +0100, Zdenek Kabelac wrote:
>> Dne 28. 01. 21 v 11:27 Martin Wilck napsal(a):
>>> On Thu, 2021-01-28 at 11:10 +0100, Zdenek Kabelac wrote:
>>>> Dne 27. 01. 21 v 18:28 mwilck at suse.com?napsal(a):
>>>>> From: Martin Wilck <mwilck@suse.com>

> 
> Of course *udev* works "in my main system". But *LVM2* does not: with
> the default setting "external_device_info_source=none", it ignores udev
> properties of devices. This is the source of lots of subtle errors and
> race conditions during device setup. Therefore we changed the setting
> to "udev".

The reason why "udev" is not default is simply because ATM it's unreliable 
source of info. It works well only for couple device types - but for lot of 
others it is not actual (i.e. network storage) (there are even no events 
generated for updating udev)

So it's purely on admins choice to select the best working path.

Internal scan in lvm2 has been 'accelerated' with usage of asynchronous
reading so in most cases it should be giving good enough performance.

So for most user selecting 'right' filter rules gives the most 'stable' results.

If we would have thought for a minute that 'udev' is good generic
default - it would be already done....

> 
> How do you handle that in Fedora? I took the liberty to look at the
> Fedora 33 package, and it doesn't change default from "none" to
> "udev".?So by common sense, Fedora is going to suffer from the same
> general problem that (open)SUSE sees: With "none", lvm can detect

Generic advice is - configure proper filters.

There will be also a new 'filtering' introduced in a form a basic acceptance 
list of devices - that may be seen in some cases as more simple to use.

(i.e. once you make 'pvcreate' - device will listed in a file of accepted
devices and lvm2 - instead of writing manual filter rules - but it's
its own set of problems...) - so skilled admins may still prefer regex filters.

> multipath or MD components only "after the fact", i.e. after multipathd
> or mdadm have grabbed them already. If pvscan and multipathd start up
> simultaneously, it's anyones guess who "wins" (*). With "udev", that
> can't happen, and that's why "udev" should be made the default.

There are many other types where udev is not capable to keep proper updated
info and there is no maintainer for those device types.

Also there is major generic trouble about complete lack of any kind of
synchronization - so as said - the info in udev DB is unfortunately
very fragile.

But you are right that mpath knowledge is the one well maintained...

There is also project SID - that might hopefully improve this and
lvm2 might be able to use this as more reliable source of info about devices.

>> So in general - this fallback should be only like new configurable
>> option - since normally you do not want lvm2 to ever touch /dev
>> dir which is under udev control.

With some hidden fallback (as well as with timeouts) there is also the problem
with 'randomness'.

So as said - if there would be a new mode with a fallback rule in its 
description - it'd be fine.

but from users POV - we can't randomly change the 'game' without giving
users chance to keep their 'old' logic which could have been more correct for 
them.

> But that would also mean that you would have to change the default to
> "udev", and *remove* both options "external_device_info_source" and
> "obtain_device_list_from_udev". The former should be hard coded to
> "udev" and the latter to 1, end of story. If you don't remove these
> options, how would the new option interact with the existing two? Which
> would take precedence?

Each - has different meaning:

obtain_device_list_from_udev:  should give us devices for scanning - this
is pretty much ok - but not much different for list of /sys/block anyway.

external_device_info_source:   delegate 'trust' to udev knowledge about device 
- which is unfortunately in many cases wrong so we cannot switch to this as 
default - as there would be too many error reports
(even our own 'pvscan' service is currently executed as asynchronous
task - so even info about lvm2 devices in udevdb can be invalid...)


> Explain that, please. The fallback does nothing in the current default
> case (external_device_info_source="none"). And in the "udev" case, it
> avoids an error condition in special situations, simply by falling back
> to the current default. What's wrong about that?

It seems to base here everything on the better udev info for offline mpath,
but there are far more case where udev is simply wrong.

If your only problem is mpath leg detection - one solution could be - to 
improve lvm2 internal mpath leg detection - there is even some initial patch 
set in some Dave's branch - which would need just some polishing to become 
probably more usable then 'udev info source'.

> Again, this is not only about containers, but any environment where
> the udev data base is not available.

For systems without 'working/usable' udev - users should reconfigure
lvm2 to not work with udev.

But there need to be seen a big difference between systems, where
udev i.e. temporarily crashed - so it may look-like a system
without udevd - and real non-udev based distro.

ATM lvm2 has no knowledge how to recognizes those 2 for all cases...

> If you can provide a better solution than my patch, we'll happily
> take it. But we need *something* to fix the current breakage.

1st. we should define what kind of problem you try to solve
and eventually open BZ (since list is not best for tracking progress).

Is is detection of 'mpath leg' while mpath is offline
and user is not able to set his filters properly ?

Zdenek


--
And I'll just repeat myself - users are NOT supposed to be using lvm2 in their 
containers and if they do so - it's generically unsupportable from lvm2 with 
current infrastructure.

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Fri, 2021-01-29 at 12:11 +0100, Zdenek Kabelac wrote:
> 
> The reason why "udev" is not default is simply because ATM it's
> unreliable 
> source of info. It works well only for couple device types - but for
> lot of 
> others it is not actual (i.e. network storage) (there are even no
> events 
> generated for updating udev)

What kind of network storage?

> If we would have thought for a minute that 'udev' is good generic
> default - it would be already done....

The udev source (DEV_EXT_UDEV) is only used in 5 places, all in the
filtering code path:

 a) dev_is_md_component()
 b) _dev_is_mpath()
 c) dev_is_partitioned()
 d) _dev_is_fwraid()
 e) check_pv_min_size()

I think we agree that "udev" is more reliable than "none" in case a)
and b). For which of the above is "udev" unreliable, why, and why can't
it be fixed?

> > How do you handle that in Fedora? 
> Generic advice is - configure proper filters.

IMHO filters aren't the solution. LVM offers
"multipath_component_detection", which suggests that
no additional filtering is necessary. Unfortunately
"multipath_component_detection" is broken without
'external_device_info_source="udev"'.

LVM filters are hard to get right, even for experienced admins. Not
because of the regexes, but because of the many different names under
which devices appear under /dev. Telling admins that they need to
fiddle with filter rules just to make multipath work sounds like
something I'd have said 10 years ago, but not in 2021.

> There will be also a new 'filtering' introduced in a form a basic
> acceptance 
> list of devices - that may be seen in some cases as more simple to
> use.

Perhaps this will improve matters, perhaps it'll add more confusion.?
I don't know enough to tell.


> > But that would also mean that you would have to change the default
> > to
> > "udev", and *remove* both options "external_device_info_source" and
> > "obtain_device_list_from_udev". The former should be hard coded to
> > "udev" and the latter to 1, end of story. If you don't remove these
> > options, how would the new option interact with the existing two?
> > Which
> > would take precedence?
> 
> Each - has different meaning:
> 
> obtain_device_list_from_udev:? should give us devices for scanning -
> this
> is pretty much ok - but not much different for list of /sys/block
> anyway.
> 
> external_device_info_source:?? delegate 'trust' to udev knowledge
> about device 

Thanks for the explanation. It makes sense, even though I think not all
combinations do (if you trust udev, why not use it for obtaining the
device list?). "lvmconfig --withcomments" is a bit vague about this.

> If your only problem is mpath leg detection - one solution could be -
> to 
> improve lvm2 internal mpath leg detection - there is even some
> initial patch 
> set in some Dave's branch - which would need just some polishing to
> become 
> probably more usable then 'udev info source'.

Which branch are you referring to?


> > If you can provide a better solution than my patch, we'll happily
> > take it. But we need *something* to fix the current breakage.
> 
> 1st. we should define what kind of problem you try to solve
> and eventually open BZ (since list is not best for tracking
> progress).
> 
> Is is detection of 'mpath leg' while mpath is offline
> and user is not able to set his filters properly ?

See above. Our aspiration is to make multipath+md+LVM setups work out-
of-the-box, reliably.

Regards,
Martin

[PATCH] config: set external_device_info_source=none if udev isn't running

[Zdenek Kabelac]

Dne 29. 01. 21 v 14:02 Martin Wilck napsal(a):
> On Fri, 2021-01-29 at 12:11 +0100, Zdenek Kabelac wrote:
>>
>> The reason why "udev" is not default is simply because ATM it's
>> unreliable
>> source of info. It works well only for couple device types - but for
>> lot of
>> others it is not actual (i.e. network storage) (there are even no
>> events
>> generated for updating udev)
> 
> What kind of network storage?
> 
>> If we would have thought for a minute that 'udev' is good generic
>> default - it would be already done....
> 
> The udev source (DEV_EXT_UDEV) is only used in 5 places, all in the
> filtering code path:
> 
>   a) dev_is_md_component()
>   b) _dev_is_mpath()
>   c) dev_is_partitioned()
>   d) _dev_is_fwraid()
>   e) check_pv_min_size()
> 
> I think we agree that "udev" is more reliable than "none" in case a)
> and b). For which of the above is "udev" unreliable, why, and why can't
> it be fixed?

Unfortunately I can't agree.

Udev ONLY works well on systems without any problems. As soon as your
devices start to be temporarily missing or you have 'duplicate' device,
the whole udevDB becomes invalid.

While with lvm2 you can filter out particular device set - with udev the
system cannot be trusted at all.

> 
>>> How do you handle that in Fedora?
>> Generic advice is - configure proper filters.
> 
> IMHO filters aren't the solution. LVM offers

With good and IMHO not so hard to configure allowance filters,
user easily solve any problems - as lvm2 will ONLY see
valid PVs and all other devices will be ignored.

IMHO every user should set 'minimal' set of visible devices to speedup all 
lvm2 commands and avoid unnecessary access to all other devices.

> "multipath_component_detection", which suggests that
> no additional filtering is necessary. Unfortunately
> "multipath_component_detection" is broken without
> 'external_device_info_source="udev"'.

It is not broken, it just only works with 'active' mpath devices.
User typically doesn't want to run lvm2 command while mpath is
not running, so problem usually happens on misconfigured system.
With running mpath - detection works.
Also note - lvm2 will usually report errors when it spots multiple legs with 
duplicate info.

So external info is only needed for systems which have mpath stopped/
disable and yet the user wants/tries to manipulate with VG  - I'd not
say this as a most common use-case.

> LVM filters are hard to get right, even for experienced admins. Not
> because of the regexes, but because of the many different names under
> which devices appear under /dev. Telling admins that they need to

Hint - only real device nodes should be used - anything based on udev created 
symlinks is by its design broken anyway (due to missing 'duplication' handling 
- aka last arrival with the pathname wins all).

> fiddle with filter rules just to make multipath work sounds like
> something I'd have said 10 years ago, but not in 2021.

Not really a fault of lvm2 here - we would like to use some
reliable source of info - but udev in our testings can't deliver
correct info even on loaded system.
There is not even existing mechanism to synchronize with udev.
So there would need to be ton of sleeps and retries everywhere...

>> 1st. we should define what kind of problem you try to solve
>> and eventually open BZ (since list is not best for tracking
>> progress).
>>
>> Is is detection of 'mpath leg' while mpath is offline
>> and user is not able to set his filters properly ?
> 
> See above. Our aspiration is to make multipath+md+LVM setups work out-
> of-the-box, reliably.

I hope you do not think we have different goals - but the acceptable
solution must be reliable - not something that works only on
shiny day with few disks and as soon as something becomes invalid, the 
returned info can be very misleading.
We just hope SID can finally move this udev to a more usable state.

But please open BZ and list cases you think are broken - and we will
see what's the best way to handle them.

But from our experience and testing - enabling udev source give us so many 
problems, we simply could not keep it as default.

Regards

Zdenek

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Fri, 2021-01-29 at 17:38 +0100, Zdenek Kabelac wrote:
> Dne 29. 01. 21 v 14:02 Martin Wilck napsal(a):
> > On Fri, 2021-01-29 at 12:11 +0100, Zdenek Kabelac wrote:
> > > 
> > > The reason why "udev" is not default is simply because ATM it's
> > > unreliable
> > > source of info. It works well only for couple device types - but
> > > for
> > > lot of
> > > others it is not actual (i.e. network storage) (there are even no
> > > events
> > > generated for updating udev)
> > 
> > What kind of network storage?
> > 
> > > If we would have thought for a minute that 'udev' is good generic
> > > default - it would be already done....
> > 
> > The udev source (DEV_EXT_UDEV) is only used in 5 places, all in the
> > filtering code path:
> > 
> > ? a) dev_is_md_component()
> > ? b) _dev_is_mpath()
> > ? c) dev_is_partitioned()
> > ? d) _dev_is_fwraid()
> > ? e) check_pv_min_size()
> > 
> > I think we agree that "udev" is more reliable than "none" in case
> > a)
> > and b). For which of the above is "udev" unreliable, why, and why
> > can't
> > it be fixed?
> 
> Unfortunately I can't agree.

That's how I interpreted your prior remark:

> > But you are right that mpath knowledge is the one well maintained...

Misunderstanding, ok.

> Udev ONLY works well on systems without any problems. As soon as your
> devices start to be temporarily missing or you have 'duplicate'
> device,
> the whole udevDB becomes invalid.

Being an active multipath developer, I know that problem quite well.
In multipath-tools, we've found ways to deal with (almost)
all this weirdness though, I'd say. Corner cases remain.

> While with lvm2 you can filter out particular device set - with udev
> the
> system cannot be trusted at all.

> > "multipath_component_detection", which suggests that
> > no additional filtering is necessary. Unfortunately
> > "multipath_component_detection" is broken without
> > 'external_device_info_source="udev"'.
> 
> It is not broken, it just only works with 'active' mpath devices.
> User typically doesn't want to run lvm2 command while mpath is
> not running, so problem usually happens on misconfigured system.
> With running mpath - detection works.

This is where I disagree. The error scenarios we observed go roughly as
follows:

 - SCSI devices are being probed
 - some SCSI disk is discovered, "add" "block" uevent follows
 - multipath is run in udev rules, decides it's an mpath leg
   -> SYSTEMD_READY=0
 - multipathd sets up a map with the new path
 - "change block" uevent for the map
 - the map is now ready for upper layers to process.?
   lvm2-pvscan at .service will be run on it
 - meanwhile, other SCSI devices have beeen detected but not fully
processed yet
 - depending on timing, the pvscan instance running for the the map
   just created can grab these SCSI devices before multipathd can
   set them up. This happens because LVM doesn't honor the
   SYSTEMD_READY property.
   
I suppose you'd reply that such a system was misconfigured because
the user should have added appropriate lvm filter rules. I agree,
if that's done, the issue can't happen. But
"multipath_component_detection" alone doesn't prevent it. That's what I
wanted to say.

> So external info is only needed for systems which have mpath
> stopped/
> disable and yet the user wants/tries to manipulate with VG? - I'd not
> say this as a most common use-case.

If this was our experience, I'd never had considered using
external_device_info_source="udev" :-)


> Not really a fault of lvm2 here - we would like to use some
> reliable source of info - but udev in our testings can't deliver
> correct info even on loaded system.
> There is not even existing mechanism to synchronize with udev.
> So there would need to be ton of sleeps and retries everywhere...

What kind of test scenario were you using?

> > > 1st. we should define what kind of problem you try to solve
> > > and eventually open BZ (since list is not best for tracking
> > > progress).
> > > 
> > > Is is detection of 'mpath leg' while mpath is offline
> > > and user is not able to set his filters properly ?
> > 
> > See above. Our aspiration is to make multipath+md+LVM setups work
> > out-
> > of-the-box, reliably.
> 
> I hope you do not think we have different goals - but the acceptable
> solution must be reliable - not something that works only on
> shiny day with few disks and as soon as something becomes invalid,
> the 
> returned info can be very misleading.

Funny you say that :-) I proposed this patch precisely because in our
experience it *improves* reliability. So, we have the same goals, just
different experimental evidence.

I start to realize that our switching to
external_device_info_source="udev" was premature.?
You certainly have more test experience with it than we do.

> We just hope SID can finally move this udev to a more usable state.

I don't know enough about SID to judge. But I think the problem is
generic. Only the kernel knows which devices are present at any given
point in time. Any user space tool can only try to be as close as
possible. SID, being more focused in scope than udev, can probably do a
better job at that, but it will still lag behind, always.

> But please open BZ and list cases you think are broken - and we will
> see what's the best way to handle them.

Hm, opening Red Hat BZs for issues that occur in SUSE Linux Enterprise
tends to be tricky. We won't usually be able to reproduce our partner
or customer issues in environments that you support. More often than
not, we can't reproduce them on our own premises. Yet they exist.

> But from our experience and testing - enabling udev source give us so
> many 
> problems, we simply could not keep it as default.

Ok, got it. I still don't quite understand why it makes my patch wrong,
but never mind.

Thanks for your time.

Martin

[PATCH] config: set external_device_info_source=none if udev isn't running

[Zdenek Kabelac]

Dne 29. 01. 21 v 18:58 Martin Wilck napsal(a):

> This is where I disagree. The error scenarios we observed go roughly as
> follows:
> 
>   - SCSI devices are being probed
>   - some SCSI disk is discovered, "add" "block" uevent follows
>   - multipath is run in udev rules, decides it's an mpath leg
>     -> SYSTEMD_READY=0
>   - multipathd sets up a map with the new path
>   - "change block" uevent for the map
>   - the map is now ready for upper layers to process.
>     lvm2-pvscan at .service will be run on it
>   - meanwhile, other SCSI devices have beeen detected but not fully
> processed yet
>   - depending on timing, the pvscan instance running for the the map
>     just created can grab these SCSI devices before multipathd can
>     set them up. This happens because LVM doesn't honor the
>     SYSTEMD_READY property.

Please open BZ with your findings.

We already know that 'pvscan' as parallel service is not a good thing -
but it's not completely trivial to place it back into udev rule.
So far there was no big priority put into this - but I think it's
causing some harm to scan as service -  we should need only
activation as service.


> I suppose you'd reply that such a system was misconfigured because
> the user should have added appropriate lvm filter rules. I agree,
> if that's done, the issue can't happen. But
> "multipath_component_detection" alone doesn't prevent it. That's what I
> wanted to say.
> 
>> So external info is only needed for systems which have mpath
>> stopped/
>> disable and yet the user wants/tries to manipulate with VG? - I'd not
>> say this as a most common use-case.
> 
> If this was our experience, I'd never had considered using
> external_device_info_source="udev" :-)
> 

One of the main problems is 'synchronization' with udev - which is simply not 
there.

So if you run commands by hand in shell - it usually does not matter - there 
is usually big enough latency.
But if commands are executed through scripts - there is big randomness
how the 'script' will work.  Devices are there. Preceding command have created 
them - but udev doesn't know about them yet, or devices are already something 
different.

If there would be a way how we can know the udev has finished
all the work with all 'already' discovered device - but since many tasks are 
now handle with asynchronous services - it's even more complicated.

> Funny you say that :-) I proposed this patch precisely because in our
> experience it *improves* reliability. So, we have the same goals, just
> different experimental evidence.

It needs to first start by check there is exactly one
udev check call for detection of disabled udev.

> I don't know enough about SID to judge. But I think the problem is
> generic. Only the kernel knows which devices are present at any given
> point in time. Any user space tool can only try to be as close as
> possible. SID, being more focused in scope than udev, can probably do a
> better job at that, but it will still lag behind, always.

We hope SID would not be confused with disks with duplicated signatures
and other weirdness common in running systems.

>> But please open BZ and list cases you think are broken - and we will
>> see what's the best way to handle them.
> 
> Hm, opening Red Hat BZs for issues that occur in SUSE Linux Enterprise
> tends to be tricky. We won't usually be able to reproduce our partner

We do have 'lvm2 community bugzilla' for lvm2 bugs. So really everyone can
open them - we just need to get logs attached for analysis and preferably
some test case.

Our experience is, that often bugs have usually quite different roots....
So we would like to first get analysis of root cause of the issue
instead of just hiding bug but some 'hidden' auto switch.

Zdenek

[PATCH] config: set external_device_info_source=none if udev isn't running

[David Teigland]

On Fri, Jan 29, 2021 at 02:02:11PM +0100, Martin Wilck wrote:
> LVM filters are hard to get right, even for experienced admins. Not
> because of the regexes, but because of the many different names under
> which devices appear under /dev. Telling admins that they need to
> fiddle with filter rules just to make multipath work sounds like
> something I'd have said 10 years ago, but not in 2021.

I agree.

> > There will be also a new 'filtering' introduced in a form a basic
> > acceptance 
> > list of devices - that may be seen in some cases as more simple to
> > use.
> 
> Perhaps this will improve matters, perhaps it'll add more confusion.?
> I don't know enough to tell.

The filter problems, among many others, were the motivation to create a
new feature "devices file" which I'm planning to merge into lvm soon.

Latest devel branch (see latest -N suffix):
https://sourceware.org/git/?p=lvm2.git;a=shortlog;h=refs/heads/dev-dct-devicesid-22

The basic idea is that a new lvm config file (the devices file) lists the
devices that lvm can use, and lvm will not even look at devices outside
that list.  So the user tells lvm which devices it should use, and that's
all lvm sees.  (With an optional allowance for pvcreate to implicitly mean
"use this device", to make it easier for people to begin using this.)

The devices are primarily identified by a WWID, or other similar
identifier.  lvm commands are used manage that file, but it can be
manually edited also.  It replaces filters, and will make a number of hard
problems in lvm just go away:  the problem of unstable dev names in
filters goes away, mpath/md component detection goes away (you wouldn't
include component devs in the device file), the system mistakenly
activating PVs belonging to VMs goes away.

This is obviously not an immediate solution to the problems you're having,
but it's what we have in mind as the best solution, available fairly soon.
For immediate fixes I still think native detection supplemented with udev
or other info is ideal.  Don't feel constrained by the old definitions of
config settings - those config settings can be superceded by new ones.

The new devices file code, even when the devices file is not enabled, will
make it easier to consume /etc/multipath/wwids (since the devices file is
using wwids itself).  So we can begin improving mpath component detection
before users enable the devices file.

Dave

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Fri, 2021-01-29 at 11:34 -0600, David Teigland wrote:
> On Fri, Jan 29, 2021 at 02:02:11PM +0100, Martin Wilck wrote:
> > LVM filters are hard to get right, even for experienced admins. Not
> > because of the regexes, but because of the many different names
> > under
> > which devices appear under /dev. Telling admins that they need to
> > fiddle with filter rules just to make multipath work sounds like
> > something I'd have said 10 years ago, but not in 2021.
> 
> I agree.
> 
> > > There will be also a new 'filtering' introduced in a form a basic
> > > acceptance 
> > > list of devices - that may be seen in some cases as more simple
> > > to
> > > use.
> > 
> > Perhaps this will improve matters, perhaps it'll add more
> > confusion.?
> > I don't know enough to tell.
> 
> The filter problems, among many others, were the motivation to create
> a
> new feature "devices file" which I'm planning to merge into lvm soon.
> 
> Latest devel branch (see latest -N suffix):
>  
> https://sourceware.org/git/?p=lvm2.git;a=shortlog;h=refs/heads/dev-dct-devicesid-22
> 

Interesting, thanks.

> The basic idea is that a new lvm config file (the devices file) lists
> the
> devices that lvm can use, and lvm will not even look at devices
> outside
> that list.? So the user tells lvm which devices it should use, and
> that's
> all lvm sees.? (With an optional allowance for pvcreate to implicitly
> mean
> "use this device", to make it easier for people to begin using this.)
> 
> The devices are primarily identified by a WWID, or other similar
> identifier.? lvm commands are used manage that file, but it can be
> manually edited also.? It replaces filters, and will make a number of
> hard
> problems in lvm just go away:? the problem of unstable dev names in
> filters goes away, mpath/md component detection goes away (you
> wouldn't
> include component devs in the device file), the system mistakenly
> activating PVs belonging to VMs goes away.

If there are both SCSI devices and multipath devices with the given
WWID, multipath would be preferred, I suppose? What would happen during
device detection, when a SCSI device with the given WWID pops up? How
would lvm "know" that this device was going to be added to a multipath
map without asking udev?

Regards
Martin

[PATCH] config: set external_device_info_source=none if udev isn't running

[David Teigland]

On Fri, Jan 29, 2021 at 08:58:21PM +0100, Martin Wilck wrote:
> If there are both SCSI devices and multipath devices with the given
> WWID, multipath would be preferred, I suppose? What would happen during
> device detection, when a SCSI device with the given WWID pops up? How
> would lvm "know" that this device was going to be added to a multipath
> map without asking udev?

Multipath devs and scsi devs are recorded as different types in the
devices file.  So, when looking for the wwid lvm will only look at mpath
devices to find a match for it.

Ideally the user does not also include the scsi component in the devices
file, but if they do lvm's existing mpath component filter is used to
ignore it.

As an example, I have /dev/mapper/mpatha over /dev/sdf.

The devices file mpatha entry is:
IDTYPE=mpath_uuid IDNAME=mpath-3690b11c0000438ad0000057150910423 DEVNAME=/dev/mapper/mpatha PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU

in this case lvm never looks at /dev/sdf.

After adding sdf to the devices file (ignoring the warnings about it):
IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000057150910423 DEVNAME=/dev/sdf PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU
IDTYPE=mpath_uuid IDNAME=mpath-3690b11c0000438ad0000057150910423 DEVNAME=/dev/mapper/mpatha PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU

in this case lvm's mpath filter ignores sdf.

"sys_wwid" entries are matched based on the /sys/.../device/wwid value and
"mpath_uuid" entries are matched based on the /sys/.../dm/uuid value.

(Note the unreliable DEVNAME values in there are used only as hints and
lvm corrects these when it finds they've become incorrect.)

Dave

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Fri, 2021-01-29 at 14:39 -0600, David Teigland wrote:
> On Fri, Jan 29, 2021 at 08:58:21PM +0100, Martin Wilck wrote:
> > If there are both SCSI devices and multipath devices with the given
> > WWID, multipath would be preferred, I suppose? What would happen
> > during
> > device detection, when a SCSI device with the given WWID pops up?
> > How
> > would lvm "know" that this device was going to be added to a
> > multipath
> > map without asking udev?
> 
> Multipath devs and scsi devs are recorded as different types in the
> devices file.? So, when looking for the wwid lvm will only look at
> mpath
> devices to find a match for it.
> 
> Ideally the user does not also include the scsi component in the
> devices
> file, but if they do lvm's existing mpath component filter is used to
> ignore it.
> 
> As an example, I have /dev/mapper/mpatha over /dev/sdf.
> 
> The devices file mpatha entry is:
> IDTYPE=mpath_uuid IDNAME=mpath-3690b11c0000438ad0000057150910423
> DEVNAME=/dev/mapper/mpatha PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU
> 
> in this case lvm never looks at /dev/sdf.
> 
> After adding sdf to the devices file (ignoring the warnings about
> it):
> IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000057150910423
> DEVNAME=/dev/sdf PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU
> IDTYPE=mpath_uuid IDNAME=mpath-3690b11c0000438ad0000057150910423
> DEVNAME=/dev/mapper/mpatha PVID=G7YCC9mQZPSvu21oWivrjM2yPtTb55dU
> 
> in this case lvm's mpath filter ignores sdf.

If it works :-)

> "sys_wwid" entries are matched based on the /sys/.../device/wwid
> value and
> "mpath_uuid" entries are matched based on the /sys/.../dm/uuid value.
> 

Nice work, looks pretty solid to me. Perhaps we could add automation to
multipath-tools for creating such entries ....

Cheers,
Martin

[PATCH] config: set external_device_info_source=none if udev isn't running

[David Teigland]

Hi Martin, thanks for the patch, it looked obviously correct on first
glance, but with this discussion perhaps I missed something, so I'll have
to go back and look again.

On Fri, Jan 29, 2021 at 11:07:12AM +0100, Martin Wilck wrote:
> Of course *udev* works "in my main system". But *LVM2* does not: with
> the default setting "external_device_info_source=none", it ignores udev
> properties of devices. This is the source of lots of subtle errors and
> race conditions during device setup. Therefore we changed the setting
> to "udev".
> 
> How do you handle that in Fedora? I took the liberty to look at the
> Fedora 33 package, and it doesn't change default from "none" to
> "udev".?So by common sense, Fedora is going to suffer from the same
> general problem that (open)SUSE sees: With "none", lvm can detect
> multipath or MD components only "after the fact", i.e. after multipathd
> or mdadm have grabbed them already. If pvscan and multipathd start up
> simultaneously, it's anyones guess who "wins" (*). With "udev", that
> can't happen, and that's why "udev" should be made the default.

You're right that it's extremely fragile, and we rely on the complex
systemd/udev/rules/etc machinery for all the components to work just
right.  Obviously they often don't (especially with many devices), and
it's a serious headache to sort out what happened, with few options to
actually fix or workaround problems.  In our experience we've managed to
get external=none to work pretty well, although not perfect.  It sounds
like it's not working quite as well in your case, which I'd guess is due
to slight differences in the udev-related machinery.  We usually end up
adjusting udev rules when problems come up.  As Zdenek has mentioned,
there are also issues with external=udev (although we don't have as much
experience to give details).  I don't think external=udev would be a big
improvement, at least in our case.  Given my level of disregard for udev
overall (trying to be polite about that), I now prefer that we do
everything we can natively, without depending on udev.  I think the best
approach is to use native detection *plus* udev info in spots, so we get
all possible info, without relying entirely on udev.  So I think the
answer is to use any/all sources of info that are available at the time we
need it, and not configure one or the other statically.

> (I'm cc'ing Ben Marzinski, as he should know this problem very well,
> and knows Fedora, too).

Quite a while ago I talked with Ben about making lvm read
/etc/multipath/wwids to detect mpath components.  I wrote an initial patch
which I never finished since there wasn't interest.  Now things have
changed, and I also have a new feature (devices file) that will make it
easier to use the mpath wwids.  Ben also mentioned a new library that we
might also look at using to get the the mpath wwids.

Dave

[PATCH] config: set external_device_info_source=none if udev isn't running

[Martin Wilck]

On Fri, 2021-01-29 at 10:59 -0600, David Teigland wrote:
> 
> Hi Martin, thanks for the patch, it looked obviously correct on first
> glance, but with this discussion perhaps I missed something, so I'll
> have
> to go back and look again.
> 
> On Fri, Jan 29, 2021 at 11:07:12AM +0100, Martin Wilck wrote:
> > Of course *udev* works "in my main system". But *LVM2* does not:
> > with
> > the default setting "external_device_info_source=none", it ignores
> > udev
> > properties of devices. This is the source of lots of subtle errors
> > and
> > race conditions during device setup. Therefore we changed the
> > setting
> > to "udev".
> > 
> > How do you handle that in Fedora? I took the liberty to look at the
> > Fedora 33 package, and it doesn't change default from "none" to
> > "udev".?So by common sense, Fedora is going to suffer from the same
> > general problem that (open)SUSE sees: With "none", lvm can detect
> > multipath or MD components only "after the fact", i.e. after
> > multipathd
> > or mdadm have grabbed them already. If pvscan and multipathd start
> > up
> > simultaneously, it's anyones guess who "wins" (*). With "udev",
> > that
> > can't happen, and that's why "udev" should be made the default.
> 
> You're right that it's extremely fragile, and we rely on the complex
> systemd/udev/rules/etc machinery for all the components to work just
> right.? Obviously they often don't (especially with many devices),
> and
> it's a serious headache to sort out what happened, with few options
> to
> actually fix or workaround problems.? In our experience we've managed
> to
> get external=none to work pretty well, although not perfect.? It
> sounds
> like it's not working quite as well in your case, which I'd guess is
> due
> to slight differences in the udev-related machinery.? We usually end
> up
> adjusting udev rules when problems come up.? As Zdenek has mentioned,
> there are also issues with external=udev (although we don't have as
> much
> experience to give details).? I don't think external=udev would be a
> big
> improvement, at least in our case.? Given my level of disregard for
> udev
> overall (trying to be polite about that), I now prefer that we do
> everything we can natively, without depending on udev.? I think the
> best
> approach is to use native detection *plus* udev info in spots, so we
> get
> all possible info, without relying entirely on udev.? So I think the
> answer is to use any/all sources of info that are available at the
> time we
> need it, and not configure one or the other statically.

I agree with almost everything you say. My line of thinking is that 
modern Linux distros are fundamentally built around udev, the main
reason being the integration of udev and systemd. The advantage of udev
is it's flexibility and the possibility to do almost everything you'd
imagine with udev rules. And that's the biggest disadvantage at the
same time, because it opens up the device handling for any nonsense.
Distros can only hope that users won't mess it up. Not to mention that
it's slow.

multipath-tools has slowly moved from "native" device detection towards
relying almost 100% on udev during the last decade.?The main motivation
were the difficulties of vendor-specific device identification that
could be handled better by the flexibility of udev rules than by
anything we could hard-code. Today, device identification is well
standardized and could well be handled natively. Ben and I already
discussed reverting back from udev to native device detection some time
ago. I doubt it'll happen very soon, though.

Maybe I should re-phrase my point like this: We need to be consistent.
We can't rely on udev here and use something else there. And that's
what lvm2 does by default today with obtain_device_list_from_udev=1 and
external_device_info_source=none.
> 

> Quite a while ago I talked with Ben about making lvm read
> /etc/multipath/wwids to detect mpath components.? I wrote an initial
> patch
> which I never finished since there wasn't interest.? Now things have
> changed, and I also have a new feature (devices file) that will make
> it
> easier to use the mpath wwids.? Ben also mentioned a new library that
> we
> might also look at using to get the the mpath wwids.

Yes, I was involved in that :-)

Martin