[linux-lvm] LVM2, NFS and random device (major:minor) numbers

[linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Paul Raines]

I have a NFS server still running CentOS 4.0 (kernel 2.6.9-5.0.5.ELsmp, 
lvm2-2.00.31-1.0.RHEL4).  It has two 3ware 9500S-8 controllers translating 
to two RAIDs at /dev/sda and /dev/sdb. The OS is on a small regular IDE 
disk.  Both sda and sdb are 2TBs, have a GPT partition table and have one 
LVM flagged partition taking up the whole 2TBs -- /dev/sda1 and /dev/sdb1 
each made into PVs.  I created two volume groups, vg1 on sda1 and vg2 on 
sda2.  No other PVs involved.

I started making volumes on vg1 and no problems.  Recently I made a couple 
volumes on vg2 and started using them with no problems.  Then yesterday I 
rebooted the box for the first time since creating volumes on vg2. 
Suddenly I get reports from my users who have NFS mounted the volumes that 
data is missing or wrong.  Looking closely I see that volumes off this 
server are suddenly mounted in the wrong spots off by two (volume1 is 
mounted where volume3 is supposed to be, volume2 is mounted where volume 4 
is supposed to be, and son on).

NFS depends on what the server says the underlying device id is of the 
exported volume and if that changes, then NFS mounts are going to change 
out from under themselves.  And everytime I add a volume and reboot the 
device ids change.

I can see that vgscan always sees vg2 before vg1 but gives no way to change 
the order it sees things.  I suspect if I had filled up vg2 first and then 
made volumes on vg1, I never would have discovered this problem unless
I went back and deleted a volume on vg2.

After frantically search the net about what the hell is going on I first 
find a Changelog for lvm1 saying a bug just like this got fixed. But this 
is lvm2 and it seems to not be fixed.  I try upgrading to CentOS 4.2 but 
testing the issue again by creating more volumes on vg2 and the rebooting 
show the underlying device ids change once again.

I finally come upon this old post to this list:

http://www.redhat.com/archives/linux-lvm/2005-May/msg00029.html

QUESTION 1: My first question is why aren't major/minor assignments 
persistent in the first place by default?  I cannot see a reason why not 
and the current default behavior is just asking for trouble for anyone 
doing NFS export of their LVM volumes.  It also screwes up many incremental 
backup programs that depend on persistent device numbers so the exports 
fsid= thing is not a full solution.

It seems like something about this should be screamed as a loud warning in 
the LVM HOWTO if things really are supposed to work way, but I could find 
no reference to it at all.

On my server, I turned off the NFS server, umounted all the volumes and 
desperately tried to think of a way I could get my volumes back to their 
original device IDs without having to possibly reboot over 500 Linux 
clients to clear the issue.  My first attempt was to use

    lvchange -My --major ### --minor ###

on my volumes. On the first volume in vg1 I did

    lvchange -My --major 253 --minor 1

and on the second I did the same with minor of 2, and so on.  Then in
vg2 I decided the easiest thing was give it a different major, so
on its first volume I did:

    lvchange -My  --major 254 --minor 1

and so on on the other five volumes in vg2.  Then reboot.  And discovered 
that my first five volumes in vg1 refused to mount complaining another
volume was already using its major/minor numbers.

Doing a 'lvdisplay -v' on the first volume in vg2, I could see it was
still using major 253 despite having given it major 254 above.  In
fact the lvdisplay showed inconsistent data like this:

   Persistent major       254
   Persistent minor       1
   Block device           253:101

QUESTION 2: I then saw in the mail post above that kernel 2.6 ignores
the major number given.  But the lvchange command insisted that I give
it anyway giving me the illusion that it mattered.  So I think this
is a bug in lvchange that should be fixed.  But is the major number
something that could still be in flux and someday I find all my
volumes have changed device number again due to the device mapper giving
all LVM volumes sam random new major?

So I ending up going to scheme where on vg2 volumes I still use major 253 
and start the minor numbers at 100+<volnum> so the first volume on vg2 has 
minor 101.  This finally seemed to work okay after reboot (assuming
the major number is never going to change).

QUESTION 3: So it seems as standard practice on using lvcreate from now on 
I need to -My to prevent these problems.  Is this what everyone else is 
doing?

QUESTION 4: I have several other severs configured just like the one above. 
And on one I know have created new volumes on the 2nd volume group and have 
not rebooted it since.  I am now afraid to do so and I am trying to figure 
out what I can do.  Can I do 'lvdisplay' on each live, NFS exported volume 
to get its current major:minor and then run 'lvchange -My' on it also live 
to set it to persistant?  Or will it have to be offline?


-- 
---------------------------------------------------------------
Paul Raines                email: raines at nmr.mgh.harvard.edu
MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging
149 (2301) 13th Street     Charlestown, MA 02129	    USA

Re: [linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Alasdair G Kergon]

On Fri, Apr 14, 2006 at 10:08:40AM -0400, Paul Raines wrote:
> I have a NFS server still running CentOS 4.0 (kernel 2.6.9-5.0.5.ELsmp, 
> lvm2-2.00.31-1.0.RHEL4).  

man exports
/fsid

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Sander Steffann]

Hi Paul,

> NFS depends on what the server says the underlying device id is of the 
> exported volume and if that changes, then NFS mounts are going to change 
> out from under themselves.  And everytime I add a volume and reboot the 
> device ids change.

Unfortunate, but true. One of the side-effects of having dynamic volumes. 
Also note that you would get similar trouble if, for example, you added a 
SCSI device with a lower SCSI-id than an existing SCSI device. /dev/sdb1 
would become /dev/sdc1, etc.

> I finally come upon this old post to this list:
> http://www.redhat.com/archives/linux-lvm/2005-May/msg00029.html

The part that can solve everything for you is:
>> These days, that's configurable.
>> See 'man exports' fsid=num.

With an explicit fsid=n for every export, you tell the NFS server to ignore 
the device-id, and use the specified fsid instead. That way you get 
consistent NFS exports, without being dependent on the underlying 
device-ids.

I have had the same trouble as you have, and this solution works great.
Sander.

Re: [linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Paul Raines]

On Fri, 14 Apr 2006, Alasdair G Kergon wrote:

> On Fri, Apr 14, 2006 at 10:08:40AM -0400, Paul Raines wrote:
>> I have a NFS server still running CentOS 4.0 (kernel 2.6.9-5.0.5.ELsmp,
>> lvm2-2.00.31-1.0.RHEL4).
>
> man exports
> /fsid

Yes, I mentioned in my email I did not considered this a viable solution
as for one it only solves the issue for NFS and not for other programs
like incremental backup that depend on a persistent device id.

Your quick short response seems to imply to me this is a common issue
and so should really be in the HOWTO and FAQ so new admins putting
up NFS servers using LVM should be pre-informed.  Especially since
is the kind of thing that will seem to work fine at the start and
then bite them in the ass later on.

Re: [linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Alasdair G Kergon]

On Fri, Apr 14, 2006 at 05:22:20PM -0400, Paul Raines wrote:
> that depend on a persistent device id.

2.4 kernel supported both persistent major and persistent minor

But the changes to do that came after 2.6 forked off and they have
never been ported to 2.6.

So the tools pretend to let you specify the major - on 2.4 this
works - but on 2.6 it's ignored by the kernel.  I expect someone 
will fix the kernel side of this some day so it works like
the 2.4 version in which device-mapper can use different major
numbers for different devices.

You should be able to specify the device-mapper major number 
at boot/module load time though, as the 'major' parameter, instead
of having a dynamic one.

Persistent minors, as you discovered, should be allocated from
the top of the range.
 
Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] LVM2, NFS and random device (major:minor) numbers

[Paul Raines]

On Fri, 14 Apr 2006, Sander Steffann wrote:
>> NFS depends on what the server says the underlying device id is of the
>> exported volume and if that changes, then NFS mounts are going to change
>> out from under themselves.  And everytime I add a volume and reboot the
>> device ids change.
>
> Unfortunate, but true. One of the side-effects of having dynamic volumes.
> Also note that you would get similar trouble if, for example, you added a
> SCSI device with a lower SCSI-id than an existing SCSI device. /dev/sdb1
> would become /dev/sdc1, etc.

True, but that is a lot more obvious to an admin when it happens because 
the ARE adding a new device to the system.  Also, ironically, lvm1 actually 
protected a user from having the device id change on them due to just 
adding a new disk.  In fact making such physical device issues not a factor 
seems to be a feature of LVM the way PVs have unique UUIDs

>> I finally come upon this old post to this list:
>> http://www.redhat.com/archives/linux-lvm/2005-May/msg00029.html
>
> The part that can solve everything for you is:
>>> These days, that's configurable.
>>> See 'man exports' fsid=num.
>
> With an explicit fsid=n for every export, you tell the NFS server to ignore
> the device-id, and use the specified fsid instead. That way you get
> consistent NFS exports, without being dependent on the underlying
> device-ids.
>

I have reasons besides NFS I need the device id to be persistant.
So I want to use the -My option and would like to know if it is safe
to do on volumes in use.

> I have had the same trouble as you have, and this solution works great.

I think this speaks poorly of LVM2 that this issue which is such a big 
change from LVM1 (which was explictly patched at one point to "fix" this 
changing device ID issue) is not more well documented.  Sure, docs about 
LVM2 do say the big change is the use of the device-mapper.  But the final 
implications of this difference such as this NFS fsid issue are by no means 
obvious.


One thing I don't understand about not being able to set the major number 
in the 2.6 kernel is that since 253 will be the major number for all 
volumes, then is 256 the maximum number of LVM volumes one can have on a 
single system?  Or will the major number increment to 254 randomly
and there is no way to control it?

[linux-lvm] Re: LVM2, NFS and random device (major:minor) numbers

[Paul Raines]

Well, I just tried out the fsid option in /etc/exports and it does not 
work.  Seems to be totally ignored.

I am CentOS 4.2 with the 2.6.9-22.ELsmp kernel and nfs-utils-1.0.6-65.EL4

I have an export now defined as such:

/local_mount/space/sake/6 \
   @all(rw,fsid=64780,async,insecure)

64780 was its device id before reboot.  After reboot, the device
id became 64781 and all the current NFS mounts on clients became
stale despite the fsid setting above.   Other exports which did
not change their underlying device id did not become stale.



-- 
---------------------------------------------------------------
Paul Raines                email: raines at nmr.mgh.harvard.edu
MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging
149 (2301) 13th Street     Charlestown, MA 02129	    USA