[linux-lvm] vgscan creation of device nodes for volume groups.

[linux-lvm] vgscan creation of device nodes for volume groups.

[Jay Weber]

Hi,

I've noticed the following and am wondering about the proper way to handle
this.

I'll try to explain by example, since it's kinda awkward.

I've created 3 volume groups.  vol0, vol1, vol2.  On these I have a
logical volume for each, same names.  Each volume group lives on a
seperate device (pv).  After creation, here's the layout for the /dev/
entries.

[root@localhost jweber]# ls -la /dev/vol0
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:21 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:18 ..
crw-r-----    1 root     disk     109,   0 Feb 17 22:15 group
brw-rw----    1 root     disk      58,   0 Feb 17 22:21 vol0
[root@localhost jweber]# ls -la /dev/vol1
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:21 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:18 ..
crw-r-----    1 root     disk     109,   1 Feb 17 22:16 group
brw-rw----    1 root     disk      58,   1 Feb 17 22:21 vol1
[root@localhost jweber]# ls -la /dev/vol2
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:21 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:18 ..
crw-r-----    1 root     disk     109,   2 Feb 17 22:18 group
brw-rw----    1 root     disk      58,   2 Feb 17 22:21 vol2


Next, I turn all the volumes off and essentially turn off lvm.  I remove
one of the devices (loop devices in this case to test this out).  I'm
removing /dev/loop1 which is the device for /dev/vol1.  I then rerun
vgscan and activate my volumes.  Below are the new /dev entries for the
volumes.

[jweber@localhost jweber]$ ls -la /dev/vol0
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:38 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:38 ..
crw-r-----    1 root     disk     109,   0 Feb 17 22:38 group
brw-rw----    1 root     disk      58,   0 Feb 17 22:38 vol0
[jweber@localhost jweber]$ ls -la /dev/vol1
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:21 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:38 ..
crw-r-----    1 root     disk     109,   1 Feb 17 22:16 group
brw-rw----    1 root     disk      58,   1 Feb 17 22:21 vol1
[jweber@localhost jweber]$ ls -la /dev/vol2
total 40
dr-xr-xr-x    2 root     root         4096 Feb 17 22:38 .
drwxr-xr-x   10 root     root        36864 Feb 17 22:38 ..
crw-r-----    1 root     disk     109,   1 Feb 17 22:38 group
brw-rw----    1 root     disk      58,   1 Feb 17 22:38 vol2
[jweber@localhost jweber]$

Note the duplicate device nodes numbers for vol1 and vol2.  This makes
sense in that the prior /dev entry for a volume is left around after
volumes are disable (rather than cleaned up).  Now, the problem I'm having
with this situation is that obviously any entries in /etc/fstab based on
the device node entries may not be proper, I could end up with the wrong
devices mounted on the wrong mount points. ;(

I gather one possible way around this would be to use a filesystem label
on the filesystem on above devices and mount via that.  But last I check,
mount via fslabel didn't work with LVM devices.

Any thoughts on this?  Much appreciated.

Re: [linux-lvm] vgscan creation of device nodes for volume groups.

[Piete Brooks]

> Note the duplicate device nodes numbers for vol1 and vol2.

Dangerous -- sounds like a case for devfs ...

> This makes sense in that the prior /dev entry for a volume is left around
> after volumes are disable (rather than cleaned up).

As in `that is how the code is' rather than `that is how the code should be'?

Without devfs, it should at least cache the minor numbers, and re-use the old 
one if there is no clash.  I guess there may be a requirement for no `gaps' in 
minor numbers, but I assume not, as volumes can be de-activated causing them.

> Now, the problem I'm having with this situation is that obviously any entries
> in /etc/fstab based on the device node entries may not be proper, I could end
> up with the wrong devices mounted on the wrong mount points. ;(

... in that if a non-existant device is mounted before the existant device 
which uses the same minor number, the latter will be mounted on the former's 
mount point, and the latter will not mount as it's already mounted.

> I gather one possible way around this would be to use a filesystem label
> on the filesystem on above devices and mount via that.  But last I check,
> mount via fslabel didn't work with LVM devices.

util-linux-2.10p has:

                /* skip entire disk (minor 0, 64, ... on ide;
                   0, 16, ... on sd) */
                /* heuristic: partition name ends in a digit */

                for(s = ptname; *s; s++);
                if (isdigit(s[-1])) {
                /*
                 * Note: this is a heuristic only - there is no reason
                 * why these devices should live in /dev.
                 * Perhaps this directory should be specifiable by option.
                 * One might for example have /devlabel with links to /dev
                 * for the devices that may be accessed in this way.
                 * (This is useful, if the cdrom on /dev/hdc must not
                 * be accessed.)
                 */

> Any thoughts on this?

If you apply the patch below and create /dev/lvm[abcd], it works.

# mount -U ea3d46d0-0821-407f-bbed-0fedfe8c4dda /mnt 
# df /mnt
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/lvmd               101698        14     96564   1% /mnt
#

If the LVM code had entries lvm0, lvm1 instead of lvma, lvmb, only the latter 
would be required -- is there a reason for not doing this ?

--- mount/mount_by_label.c	Thu Oct  5 01:32:24 2000
+++ mount/mount_by_label.c	Sun Feb 18 09:00:33 2001
@@ -127 +127 @@
-		if (isdigit(s[-1])) {
+		if (isdigit(s[-1]) || !strncmp(ptname, "lvm", 3)) {

Re: [linux-lvm] vgscan creation of device nodes for volume groups.

[Jay Weber]

On Sun, 18 Feb 2001, Piete Brooks wrote:

> > Note the duplicate device nodes numbers for vol1 and vol2.
>
> Dangerous -- sounds like a case for devfs ...

Well, that's one solution. :)

> > This makes sense in that the prior /dev entry for a volume is left around
> > after volumes are disable (rather than cleaned up).
>
> As in `that is how the code is' rather than `that is how the code should be'?

Correct.  I'm not sure what the preferred, standards, or such method is
and that's exactly why I brought it up.

> Without devfs, it should at least cache the minor numbers, and re-use the old
> one if there is no clash.  I guess there may be a requirement for no `gaps' in
> minor numbers, but I assume not, as volumes can be de-activated causing them.

Right, the current in place code is proper, as I gather you don't want the
gaps.  But then you have the de-activated volume scenario and I gather you
don't want the incorrect volume pointer also.

> > Now, the problem I'm having with this situation is that obviously any entries
> > in /etc/fstab based on the device node entries may not be proper, I could end
> > up with the wrong devices mounted on the wrong mount points. ;(
>
> ... in that if a non-existant device is mounted before the existant device
> which uses the same minor number, the latter will be mounted on the former's
> mount point, and the latter will not mount as it's already mounted.

Right, it's misleading in that the prior offlined volume is what is shown
as being mounted, when in actuality it is the volume following the
offlined volume that is truely mounted.

[root@localhost /root]# mount
/dev/hda2 on / type ext3 (rw,check=none)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda3 on /opt type ext3 (rw,check=none)
/dev/vol0/vol0 on /vol0 type ext2 (rw)
/dev/vol1/vol1 on /vol1 type ext2 (rw)

Ie.. in this case /dev/vol1/vol1 on /vol1 is REALLY /dev/vol2/vol2 which
I'd normally mount on say /vol2, since I removed the device that
/dev/vol1/vol1 resided on.

> > I gather one possible way around this would be to use a filesystem label
> > on the filesystem on above devices and mount via that.  But last I check,
> > mount via fslabel didn't work with LVM devices.
>
> util-linux-2.10p has:
>
>                 /* skip entire disk (minor 0, 64, ... on ide;
>                    0, 16, ... on sd) */
>                 /* heuristic: partition name ends in a digit */
>
>                 for(s = ptname; *s; s++);
>                 if (isdigit(s[-1])) {
>                 /*
>                  * Note: this is a heuristic only - there is no reason
>                  * why these devices should live in /dev.
>                  * Perhaps this directory should be specifiable by option.
>                  * One might for example have /devlabel with links to /dev
>                  * for the devices that may be accessed in this way.
>                  * (This is useful, if the cdrom on /dev/hdc must not
>                  * be accessed.)
>                  */

Hmm, I think I like the idea of just having a common dir under /dev in
which to place all LVM devices.  Then add that dir to the mount source for
inclusion in the label scanning and resolve to mounting by label or uuid.

Using the source comments above which suggest the use of a /devlabel
dir to register label capable devices in seems interesting, but isn't that
along the lines of devfs, which would probably be a better solution in
that case?  Note, I've played with little devfs as of yet.

> If you apply the patch below and create /dev/lvm[abcd], it works.
>
> # mount -U ea3d46d0-0821-407f-bbed-0fedfe8c4dda /mnt
> # df /mnt
> Filesystem           1k-blocks      Used Available Use% Mounted on
> /dev/lvmd               101698        14     96564   1% /mnt
> #
>
> If the LVM code had entries lvm0, lvm1 instead of lvma, lvmb, only the latter
> would be required -- is there a reason for not doing this ?

Yep, I'd experimented with this in the past, but it limits you to the one
name, no good.  Using a subdir might be better in that you could then add
code to scan the entire subdir full of lvm devices.  I recall the LVM
folks have recently been addressing the issue of allowing such in recent
cvs work, maybe it's time to test it out. :)

I'd say the reason for not doing this is that LVM grants the user the
choice to name their volumes anything they like.  So in the supplied patch
if the device's name doesn't fall within your /dev/lvm[abc] naming scheme
it'll never even get scanned.

The trick is to find a solution we can all use, I'm reluctant to adopt
/dev/subdir approach just for myself and having to maintain a seperate
mount and LVM code tree which addresses such and I'm sure others are
reluctant to have to use a /dev/subdir approach also. :)

..

Re: [linux-lvm] vgscan creation of device nodes for volume groups.

[Heinz J. Mauelshagen]

On Sat, Feb 17, 2001 at 10:24:38PM -0800, Jay Weber wrote:
> Hi,
> 
> I've noticed the following and am wondering about the proper way to handle
> this.
> 

A fast workaround (assuming bash) to avoid those invalid entries
in /dev/ would be:

(cd /dev;ls -A /etc/lvmtab.d/|xargs rm -r)

You need to run it *before* vgscan for sure, because vgscan starts with
removing /etc/lvmtab*.

One extension could be to let vgscan do the removal job itself.

> I'll try to explain by example, since it's kinda awkward.
> 
> I've created 3 volume groups.  vol0, vol1, vol2.  On these I have a

<SNIP>

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] vgscan creation of device nodes for volume groups.

[Andreas Dilger]

Jay Weber writes:
> [snip]
> Next, I turn all the volumes off and essentially turn off lvm.  I remove
> one of the devices (loop devices in this case to test this out).  I'm
> removing /dev/loop1 which is the device for /dev/vol1.  I then rerun
> vgscan and activate my volumes.  Below are the new /dev entries for the
> volumes.
> [snip]
> Note the duplicate device nodes numbers for vol1 and vol2.  This makes
> sense in that the prior /dev entry for a volume is left around after
> volumes are disable (rather than cleaned up).  Now, the problem I'm having
> with this situation is that obviously any entries in /etc/fstab based on
> the device node entries may not be proper, I could end up with the wrong
> devices mounted on the wrong mount points. ;(

One of the problems of the current LVM is that it deletes and re-creates
the device nodes each time you run vgscan.  IMHO, we should keep devices
around until we do a vgexport or vgremove.  Maybe the tools can be made
so that we do not advocate the use of vgscan except in extreme failure
cases.

Otherwise, we have problems as yours, problems with NFS servers that crash,
etc.  Having non-static device numbers makes it impossible to set up NFS
server failover and other things that export device numbers from the server.

> I gather one possible way around this would be to use a filesystem label
> on the filesystem on above devices and mount via that.  But last I check,
> mount via fslabel didn't work with LVM devices.

I'm surprised that mount-by-label doesn't work.  I use this on most of my
systems, but maybe not in conjunction with LVM.  I would think the right
thing to do (rather than rely on device names) is to use device numbers
for LVM.  It may need a bit of hacking to mount, but I was actually going
to do this anyways - I want to make a "UUID and LABEL" library, because
this code is common to mount, fsck, and others.  We may want to add it to
the kernel as well (I have a patch to mount root by LABEL or UUID).  Also,
when the reiserfs folks get around to adding LABEL and/or UUID support to
their superblock, it makes sense to only need to add this to one place.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert