[BUG] multipath-tools: uuid has become meaningless

[BUG] multipath-tools: uuid has become meaningless

[Malahal Naineni]

Hi All,

	I have noticed that uuid reported my 'multipath -l' doesn't
really correspond to the paths it loaded. In fact, the machine has two
different storage subsystems, say EMC and IBM, and the 'multipath -l'
output looks like it reported EMC instead of IBM as vendor and vice
versa. Further debugging showed that the system is configured with
'user_friendly_names' in initrd.  Initrd actually configures all the
paths correctly using 'user_friendly_names' as intended, but the real
root's multipathd init script's invocation tried to match the names in
/var/lib/multipath/bindings file (this file has grown since the last
mkinitrd!).  Instead of just renaming, it actually re-loads with
different paths!

One easy way to reproduce the problem is to run multipath once, edit the
bindings file to swap couple entries (mpatha to mpathb and vice versa)
and then run multipath. The later run of multipath reloads mpatha with
mpathb's paths and vice versa, instead of just renaming. I know renaming
is hard here as they have to be unique when you rename!

Here is the code where the things happen:
libmultipath/configure.c: select_action(): This may reload if it finds
cmpp "by alias" as well as "by wwid".

Technically, this is not wrong! No I/O should be happening in the
original reported case when the device mapper tables are reloaded. It is
just that the uuid's loaded have become meaningless for user!
'multipath -l' has no point in reporting the uuid if it becomes useless!

My thoughts on fixing this:
1. Technically nothing wrong. Live with it and make sure that device
   mapper's uuid are meaningless for user and fix 'multipath -l' to
   not print uuid's.
2. Don't support user_friendly_names in initrd. Could be just documented
   or an option to multipath is added to ignore that feature and that
   option is used in initrd calls!
3. We could rename the devices instead of reload -- really fixing this!

Any comments on which way we should go and other possibilities?

Thanks, Malahal.

Re: [BUG] multipath-tools: uuid has become meaningless

[Christophe Varoqui]

[-- Attachment #1.1: Type: text/plain, Size: 958 bytes --]


> 
> My thoughts on fixing this:
> 1. Technically nothing wrong. Live with it and make sure that device
>       mapper's uuid are meaningless for user and fix 'multipath -l' to
>       not print uuid's.

I don't like this one : the uuid is the communication medium with storage teams in big IT departement.

> 2. Don't support user_friendly_names in initrd. Could be just documented
>       or an option to multipath is added to ignore that feature and that
>       option is used in initrd calls!

I let distributors comment on this one, but I guess it will replace a confusion (a naming file in initrd the sysadmin have to keep in sync) by another (root kparm not consistent with what you see on a booted system). I personnaly agree that user_friendly_names disabled in initrd would be a safer confuser.

> 3. We could rename the devices instead of reload -- really fixing this!
> 

This one seems safe.

Regards,
cvaroqui

[-- Attachment #1.2: Type: text/html, Size: 1410 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [BUG] multipath-tools: uuid has become meaningless

[Malahal Naineni]

Christophe Varoqui [christophe.varoqui@gmail.com] wrote:
>    > 3. We could rename the devices instead of reload -- really fixing this!
>    >
> 
>    This one seems safe.

This is where I was heading. I did some investigation and found that we
really need to rename to wwid based names before we finally rename to
the intended name. So we may actually have two passes just to rename
(first to wwid based names and then to actual names).  Also the current
code doesn't do rename followed by reload even if it is needed. Someone
sets friendly name and some other defaults that need reload like path
grouping policy etc, we just rename! Probably a bug in itself.

Two pass based coalesce may be fix the latter bug too.

Thanks, Malahal.
PS: I will probably post a patch and that may invite people to suggest a
better and easier method!

Re: [BUG] multipath-tools: uuid has become meaningless

[Hannes Reinecke]

Malahal Naineni wrote:
> Christophe Varoqui [christophe.varoqui@gmail.com] wrote:
>>    > 3. We could rename the devices instead of reload -- really fixing this!
>>    >
>>
>>    This one seems safe.
> 
> This is where I was heading. I did some investigation and found that we
> really need to rename to wwid based names before we finally rename to
> the intended name. So we may actually have two passes just to rename
> (first to wwid based names and then to actual names).  Also the current
> code doesn't do rename followed by reload even if it is needed. Someone
> sets friendly name and some other defaults that need reload like path
> grouping policy etc, we just rename! Probably a bug in itself.
> 
You sure? I would rather rely on device-mapper uuids here.
Point is that we
a) use 'mpath-<wwid>' as uuid for multipath generated tables anyway
and
b) uuids are a primary key for device-mapper, too.

So we can use the known naming scheme for the uuid to regenerate the
wwid; with this we can identify/access the device and we should be able to
make do with just one pass.

Two passes have the serious disadvantage of generating yet another set
of meaningless uevents, which then again have to be handled/ignored etc.

I'll be sending a patch to simplify dm_get_name() to illustrate my point.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: [BUG] multipath-tools: uuid has become meaningless

[Malahal Naineni]

Hannes Reinecke [hare@suse.de] wrote:
> Malahal Naineni wrote:
> > This is where I was heading. I did some investigation and found that we
> > really need to rename to wwid based names before we finally rename to
> > the intended name. So we may actually have two passes just to rename
> > (first to wwid based names and then to actual names).  Also the current
> > code doesn't do rename followed by reload even if it is needed. Someone
> > sets friendly name and some other defaults that need reload like path
> > grouping policy etc, we just rename! Probably a bug in itself.

> You sure? I would rather rely on device-mapper uuids here.

Hannes, I am not sure what you are referring to when you say, "You
sure?" Are you saying if I am sure about this bug? If so, you can see
yourself by swapping couple entries in the bindings file. As I said,
you will notice this bug only when you enable 'user_friendly_names' in
initrd in real life!

> So we can use the known naming scheme for the uuid to regenerate the
> wwid; with this we can identify/access the device and we should be able to
> make do with just one pass.

Say, you have "mpath0[uuid0]" and "mpath1[uuid1]" loaded currently.  How
can we rename mpath0 to mpath1 and vice-versa? It does require an
intermediate step of renaming to something else, right? 
 
Thanks, Malahal.

Re: [BUG] please PLEASE UnSubscribe!!!

[W S]

[-- Attachment #1.1: Type: text/plain, Size: 1760 bytes --]

please PLEASE UnSubscribe!!!

--- On Tue, 7/20/10, Malahal Naineni <malahal@us.ibm.com> wrote:

From: Malahal Naineni <malahal@us.ibm.com>
Subject: Re: [dm-devel] [BUG] multipath-tools: uuid has become meaningless
To: dm-devel@redhat.com
Date: Tuesday, July 20, 2010, 3:06 PM

Hannes Reinecke [hare@suse.de] wrote:
> Malahal Naineni wrote:
> > This is where I was heading. I did some investigation and found that we
> > really need to rename to wwid based names before we finally rename to
> > the intended name. So we may actually have two passes just to rename
> > (first to wwid based names and then to actual names).  Also the current
> > code doesn't do rename followed by reload even if it is needed. Someone
> > sets friendly name and some other defaults that need reload like path
> > grouping policy etc, we just rename! Probably a bug in itself.

> You sure? I would rather rely on device-mapper uuids here.

Hannes, I am not sure what you are referring to when you say, "You
sure?" Are you saying if I am sure about this bug? If so, you can see
yourself by swapping couple entries in the bindings file. As I said,
you will notice this bug only when you enable 'user_friendly_names' in
initrd in real life!

> So we can use the known naming scheme for the uuid to regenerate the
> wwid; with this we can identify/access the device and we should be able to
> make do with just one pass.

Say, you have "mpath0[uuid0]" and "mpath1[uuid1]" loaded currently.  How
can we rename mpath0 to mpath1 and vice-versa? It does require an
intermediate step of renaming to something else, right? 
 
Thanks, Malahal.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel



      

[-- Attachment #1.2: Type: text/html, Size: 2431 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [BUG] multipath-tools: uuid has become meaningless

[Hannes Reinecke]

Malahal Naineni wrote:
> Hannes Reinecke [hare@suse.de] wrote:
>> Malahal Naineni wrote:
>>> This is where I was heading. I did some investigation and found that we
>>> really need to rename to wwid based names before we finally rename to
>>> the intended name. So we may actually have two passes just to rename
>>> (first to wwid based names and then to actual names).  Also the current
>>> code doesn't do rename followed by reload even if it is needed. Someone
>>> sets friendly name and some other defaults that need reload like path
>>> grouping policy etc, we just rename! Probably a bug in itself.
> 
>> You sure? I would rather rely on device-mapper uuids here.
> 
> Hannes, I am not sure what you are referring to when you say, "You
> sure?" Are you saying if I am sure about this bug? If so, you can see
> yourself by swapping couple entries in the bindings file. As I said,
> you will notice this bug only when you enable 'user_friendly_names' in
> initrd in real life!
> 
Well, I wanted to inquire about the double renaming thing.
But

>> So we can use the known naming scheme for the uuid to regenerate the
>> wwid; with this we can identify/access the device and we should be able to
>> make do with just one pass.
> 
> Say, you have "mpath0[uuid0]" and "mpath1[uuid1]" loaded currently.  How
> can we rename mpath0 to mpath1 and vice-versa? It does require an
> intermediate step of renaming to something else, right? 
>  
is a valid point. So yes, we have to put in a renaming mechanism.
So falling back to wwid should work here.

Sorry for the noise, you are correct here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: [BUG] multipath-tools: uuid has become meaningless

[Malahal Naineni]

Christophe Varoqui [christophe.varoqui@gmail.com] wrote:
>    >
>    > My thoughts on fixing this:
>    > 1. Technically nothing wrong. Live with it and make sure that device
>    >       mapper's uuid are meaningless for user and fix 'multipath -l' to
>    >       not print uuid's.
> 
>    I don't like this one : the uuid is the communication medium with storage
>    teams in big IT departement.
> 
>    > 2. Don't support user_friendly_names in initrd. Could be just documented
>    >       or an option to multipath is added to ignore that feature and that
>    >       option is used in initrd calls!
> 
>    I let distributors comment on this one, but I guess it will replace a
>    confusion (a naming file in initrd the sysadmin have to keep in sync) by
>    another (root kparm not consistent with what you see on a booted system).
>    I personnaly agree that user_friendly_names disabled in initrd would be a
>    safer confuser.
> 
>    > 3. We could rename the devices instead of reload -- really fixing this!
>    >
> 
>    This one seems safe.

Sorry for the delay. I actually found another easier way to fix it!
Looks like the bindings file in initramfs and the active root file
system may not be in *sync* but not *inconsistent*. The problem happens
because the initramfs is mounted read/write, and multipath modifies
bindings file in memory at initramfs time that may become *inconsistent*
with the bindings file in the active root file system. 

The patch in a next mail will add an option to 'multipath' to not update
the bindings file. It will treat it as read-only, so devices that don't
have an entry in the bindings file will get WWID based names.

Thanks, Malahal.