Bug? Determining the holders of partitions shadowed by dm-multipath.

Bug? Determining the holders of partitions shadowed by dm-multipath.

[Joel Becker]

Folks,
	I think this is a bug.
I	I have a device with two paths, sda and sdb.  The device is
partitioned into two partitions (sda1, sdb1) and (sda2, sdb2).  The
multipath software will create me the devices mpath1, mpath1p1, and
mpath1p2.  So far so good.
	Now, I need to detect disks used by my software.  There is a
signature on disks used by my software, so the basic algorithm is "for
each disk, check for the signature".  But, as you know, multipath makes
this fun.  You don't want to detect sda1 or sdb1, you want to detect
mpath1.  This is true no matter how the devices are arranged (LVM
devices, kpartx of loopbacks, whatever it is).  I'll use the term
"toplevel device" to describe a device that is not being used as part
of some other device.
	So, how to determine toplevel devices?  Well,
/sys/block/.../holders works for that, so I tried it.  Sure enough,
/sys/block/sda/holders and /sys/block/sdb/holders point to mpath1.  So
my software can reliably ignore sda and sdb.  But when I look at
/sys/block/sda1/holders, it is empty.
	The disk encompassing sda1 is held by another device.  By
extension, sda1 is held.  So I think it is a bug that
/sys/block/sda1/holders is empty.
	Bug or not, what is the recommended way to determine that
/dev/sda1 should not be used, because you should be using
/dev/mpath/mpath1p1 (or whatever udev name it appears as)?

Joel

-- 

Life's Little Instruction Book #510

	"Count your blessings."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

RE: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Romanowski, John (OFT)]

my sles10sp2 at 2.6.16.60-0.29-default doesn't have the /sys/block/.../holders but
I've used major:minor device numbers from commands
dmsetup ls
and
dmsetup deps
to resolve such device relationships.

A 'dmsetup ls' device number that isn't listed in 'dmsetup deps' device numbers is a top-level device. Example below: mproot-part1 (dev 253:3, a partition of mp device 253:2) isn't listed as a dependency of any device. Likewise each listed LVM volume isn't listed as a dependency of any device; example- rootvg-tmp 253:10

pz3vf129:~ # dmsetup ls
rootvg-tmp      (253, 10)
mpboot  (253, 0)
rootvg-usr      (253, 11)
rootvg-var      (253, 12)
rootvg-opt.tivoli       (253, 7)
mproot  (253, 2)
rootvg-opt.tivoli.tsm   (253, 8)
rootvg-home     (253, 5)
mpboot-part1    (253, 1)
rootvg-opt      (253, 6)
mproot-part2    (253, 4)
rootvg-swap1    (253, 9)
mproot-part1    (253, 3)

pz3vf129:~ # dmsetup deps
rootvg-tmp: 1 dependencies      : (253, 4)
mpboot: 8 dependencies  : (8, 16) (8, 80) (8, 144) (8, 208) (8, 48) (8, 112) (8, 176) (8, 240)
rootvg-usr: 1 dependencies      : (253, 4)
rootvg-var: 1 dependencies      : (253, 4)
rootvg-opt.tivoli: 1 dependencies       : (253, 4)
mproot: 8 dependencies  : (8, 32) (8, 96) (8, 160) (8, 224) (8, 0) (8, 64) (8, 128) (8, 192)
rootvg-opt.tivoli.tsm: 1 dependencies   : (253, 4)
rootvg-home: 1 dependencies     : (253, 4)
mpboot-part1: 1 dependencies    : (253, 0)
rootvg-opt: 1 dependencies      : (253, 4)
mproot-part2: 1 dependencies    : (253, 2)
rootvg-swap1: 1 dependencies    : (253, 4)
mproot-part1: 1 dependencies    : (253, 2)



> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On Behalf Of Joel Becker
> Sent: Monday, February 09, 2009 7:50 PM
> To: dm-devel@redhat.com
> Cc: Sunil Mushran
> Subject: [dm-devel] Bug? Determining the holders of partitions shadowed
> by dm-multipath.
>
> Folks,
>       I think this is a bug.
> I     I have a device with two paths, sda and sdb.  The device is
> partitioned into two partitions (sda1, sdb1) and (sda2, sdb2).  The
> multipath software will create me the devices mpath1, mpath1p1, and
> mpath1p2.  So far so good.
>       Now, I need to detect disks used by my software.  There is a
> signature on disks used by my software, so the basic algorithm is "for
> each disk, check for the signature".  But, as you know, multipath makes
> this fun.  You don't want to detect sda1 or sdb1, you want to detect
> mpath1.  This is true no matter how the devices are arranged (LVM
> devices, kpartx of loopbacks, whatever it is).  I'll use the term
> "toplevel device" to describe a device that is not being used as part
> of some other device.
>       So, how to determine toplevel devices?  Well,
> /sys/block/.../holders works for that, so I tried it.  Sure enough,
> /sys/block/sda/holders and /sys/block/sdb/holders point to mpath1.  So
> my software can reliably ignore sda and sdb.  But when I look at
> /sys/block/sda1/holders, it is empty.
>       The disk encompassing sda1 is held by another device.  By
> extension, sda1 is held.  So I think it is a bug that
> /sys/block/sda1/holders is empty.
>       Bug or not, what is the recommended way to determine that
> /dev/sda1 should not be used, because you should be using
> /dev/mpath/mpath1p1 (or whatever udev name it appears as)?
>
> Joel
>
> --
>
> Life's Little Instruction Book #510
>
>       "Count your blessings."
>
> Joel Becker
> Principal Software Developer
> Oracle
> E-mail: joel.becker@oracle.com
> Phone: (650) 506-8127
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel


This e-mail, including any attachments, may be confidential, privileged or otherwise legally protected. It is intended only for the addressee. If you received this e-mail in error or from someone who was not authorized to send it to you, do not disseminate, copy or otherwise use this e-mail or its attachments.  Please notify the sender immediately by reply e-mail and delete the e-mail from your system.

Re: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Joel Becker]

On Tue, Feb 10, 2009 at 09:49:58AM -0500, Romanowski, John (OFT) wrote:
> my sles10sp2 at 2.6.16.60-0.29-default doesn't have the /sys/block/.../holders but
> I've used major:minor device numbers from commands
> dmsetup ls
> and
> dmsetup deps
> to resolve such device relationships.
> 
> A 'dmsetup ls' device number that isn't listed in 'dmsetup deps' device numbers is a top-level device. Example below: mproot-part1 (dev 253:3, a partition of mp device 253:2) isn't listed as a dependency of any device. Likewise each listed LVM volume isn't listed as a dependency of any device; example- rootvg-tmp 253:10

	Sure, but a /dev/sda2 won't show up in dmsetup output anywhere.
That's kind of my point.

Joel

-- 

Life's Little Instruction Book #198

	"Feed a stranger's expired parking meter."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

RE: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Romanowski, John (OFT)]

good point. sorry.
The partition's parent's device number will be in the dmsetup deps list.
/dev/sda2 is a partition; dmsetup deps will list its parent's, /dev/sda, major:minor as a dependency of the mpath device .

kludgy.

> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On Behalf Of Joel Becker
> Sent: Tuesday, February 10, 2009 4:33 PM
> To: device-mapper development
> Subject: Re: [dm-devel] Bug? Determining the holders of partitions
> shadowed by dm-multipath.
>
> On Tue, Feb 10, 2009 at 09:49:58AM -0500, Romanowski, John (OFT) wrote:
> > my sles10sp2 at 2.6.16.60-0.29-default doesn't have the
> /sys/block/.../holders but
> > I've used major:minor device numbers from commands
> > dmsetup ls
> > and
> > dmsetup deps
> > to resolve such device relationships.
> >
> > A 'dmsetup ls' device number that isn't listed in 'dmsetup deps'
> device numbers is a top-level device. Example below: mproot-part1 (dev
> 253:3, a partition of mp device 253:2) isn't listed as a dependency of
> any device. Likewise each listed LVM volume isn't listed as a
> dependency of any device; example- rootvg-tmp 253:10
>
>       Sure, but a /dev/sda2 won't show up in dmsetup output anywhere.
> That's kind of my point.
>
> Joel
>
> --
>
> Life's Little Instruction Book #198
>
>       "Feed a stranger's expired parking meter."
>
> Joel Becker
> Principal Software Developer
> Oracle
> E-mail: joel.becker@oracle.com
> Phone: (650) 506-8127
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel


This e-mail, including any attachments, may be confidential, privileged or otherwise legally protected. It is intended only for the addressee. If you received this e-mail in error or from someone who was not authorized to send it to you, do not disseminate, copy or otherwise use this e-mail or its attachments.  Please notify the sender immediately by reply e-mail and delete the e-mail from your system.

Re: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Hannes Reinecke]

Hi Joel,

Joel Becker wrote:
> On Tue, Feb 10, 2009 at 09:49:58AM -0500, Romanowski, John (OFT) wrote:
>> my sles10sp2 at 2.6.16.60-0.29-default doesn't have the /sys/block/.../holders but
>> I've used major:minor device numbers from commands
>> dmsetup ls
>> and
>> dmsetup deps
>> to resolve such device relationships.
>>
>> A 'dmsetup ls' device number that isn't listed in 'dmsetup deps' device numbers is a top-level device. Example below: mproot-part1 (dev 253:3, a partition of mp device 253:2) isn't listed as a dependency of any device. Likewise each listed LVM volume isn't listed as a dependency of any device; example- rootvg-tmp 253:10
> 
> 	Sure, but a /dev/sda2 won't show up in dmsetup output anywhere.
> That's kind of my point.
> 
Well, yes.
That's because you have _two_ parent-child relationship in sysfs.

The one (as you already noted) is the holders / slaves directories in sysfs.
However, partitions are handled differently; they are listed under

/sys/block/sda/sda2

(note the subdir) and do not appear under holders / slaves. So the correct algorithm
is to
a) check the holders / slaves 
and
b) check if the parent directory starts with the same name and if so, check for
the holders / slaves relationship there.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Re: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Joel Becker]

On Wed, Feb 11, 2009 at 08:32:20AM +0100, Hannes Reinecke wrote:
> Hi Joel,
>
> Joel Becker wrote:
>> On Tue, Feb 10, 2009 at 09:49:58AM -0500, Romanowski, John (OFT) wrote:
>>> my sles10sp2 at 2.6.16.60-0.29-default doesn't have the /sys/block/.../holders but
>>> I've used major:minor device numbers from commands
>>> dmsetup ls
>>> and
>>> dmsetup deps
>>> to resolve such device relationships.
>>>
>>> A 'dmsetup ls' device number that isn't listed in 'dmsetup deps' device numbers is a top-level device. Example below: mproot-part1 (dev 253:3, a partition of mp device 253:2) isn't listed as a dependency of any device. Likewise each listed LVM volume isn't listed as a dependency of any device; example- rootvg-tmp 253:10
>>
>> 	Sure, but a /dev/sda2 won't show up in dmsetup output anywhere.
>> That's kind of my point.
>>
> Well, yes.
> That's because you have _two_ parent-child relationship in sysfs.
>
> The one (as you already noted) is the holders / slaves directories in sysfs.
> However, partitions are handled differently; they are listed under
>
> /sys/block/sda/sda2
>
> (note the subdir) and do not appear under holders / slaves. So the correct algorithm
> is to
> a) check the holders / slaves
> and
> b) check if the parent directory starts with the same name and if so, check for
> the holders / slaves relationship there.

	Why not just setup holders in sda2?  It's held, after all.
It's just very counterintuitive to look at an empty holders and discover
that it's empty yet held.
	Btw, do we absolutely guarantee that all block devices, forever,
will have partitions in sysfs named
"/sys/block/<parent>/<parent><more>"?

Joel

-- 

Life's Little Instruction Book #139

	"Never deprive someone of hope; it might be all they have."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: Bug? Determining the holders of partitions shadowed by dm-multipath.

[Hannes Reinecke]

Hi Joel,

Joel Becker wrote:
> On Wed, Feb 11, 2009 at 08:32:20AM +0100, Hannes Reinecke wrote:
>> Hi Joel,
>>
[ .. ]
>> Well, yes.
>> That's because you have _two_ parent-child relationship in sysfs.
>>
>> The one (as you already noted) is the holders / slaves directories in sysfs.
>> However, partitions are handled differently; they are listed under
>>
>> /sys/block/sda/sda2
>>
>> (note the subdir) and do not appear under holders / slaves. So the correct algorithm
>> is to
>> a) check the holders / slaves
>> and
>> b) check if the parent directory starts with the same name and if so, check for
>> the holders / slaves relationship there.
> 
> 	Why not just setup holders in sda2?  It's held, after all.
> It's just very counterintuitive to look at an empty holders and discover
> that it's empty yet held.

We don't need to, as sda2 is by definition held by sda.
So adding a 'holders' directory here which just points to the parent
directory is quite pointless.
(And it will become even more absurd for the 'slaves' directory in sda ...)

Remember, partitions precede the holders / slaves abstraction by years.
The former abstraction was only introduced to map device-mapper / md
device relationships onto sysfs.

> 	Btw, do we absolutely guarantee that all block devices, forever,
> will have partitions in sysfs named
> "/sys/block/<parent>/<parent><more>"?
> 
Yes, quite so. This is one relationship I would _not_ touch.

However, I'm all for it to optionally disable the partitions detection
within the kernel and have it all done in userspace by kpartx.
Then we'll be getting the 'holders/slave' relationship automatically
and won't be bothered by a separate partition scheme anymore.

Looks like it's time to dig out the patch again.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)