[linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Chris Worley]

If you remove an lv, and the minor number of the removed lv is less than
the minor number of your root partition, then the vgscan during initrd
will run correctly, but assign a minor device number to the root
partition that lilo doesn't expect (and you won't be able to mount the
root partition).

This caused a few tense moments with root booting LVM this morning ;)...

With a new big hard drive in hand, I started upgrading my server.  Added
the new disk as it's own logical volume, reiser format, root initrd'd,
and copied all my other disks to it, then started upgrading to SuSE 7.3.

During the upgrade, I was running out of disk space , so I decommisioned
one of the older lv's with lvremove, then lvextended it onto the new lv,
and had reiser expand into the new space.  Quite easy and nifty; that's
what LVM's all about.

I even ran lilo before the next reboot, but it wouldn't have mattered...

Upon the next reboot, the initrd ran fine... found and activated the
lv's... but the kernel said "no root partition" and hung.  I could
rescue boot the system from the SuSE CD -- everything was there... I
finally added a "vgdisplay -v" to the initrd's linuxrc, and that showed
me the problem.

The file system it was trying to mount was, to lilo, the device "3a:03"
(major device number 58, minor 3), but, once I'd removed the old lv for
extra disk space, any new vgscan during initrd assigned my new root lv
to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
doesn't like the name).  There was no longer an lvm minor device 3 to
mount.

Once I saw the problem, I recue booted and chrooted to my new lvm, and
manually changed the minor number before running lilo.  

I'm guessing vgscan can be run on an active system to realign the minor
numbers, before running lilo, but I'm not sure... 

_BUT_, I think the faq's ought to add a little note concerning this side
effect to root booted lvm file systems to the "Removing a logical
volume" section, and save the next person some "tense moments ;)".

If this info is already there, then disregard this sugestion (I haven't
seen it, but I do tend to skim the manuals).

Chris

RE: [linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Ben Holness]

> Upon the next reboot, the initrd ran fine... found and activated the
> lv's... but the kernel said "no root partition" and hung.  I could
> rescue boot the system from the SuSE CD -- everything was there... I
> finally added a "vgdisplay -v" to the initrd's linuxrc, and
> that showed me the problem.
>
> The file system it was trying to mount was, to lilo, the device "3a:03"
> (major device number 58, minor 3), but, once I'd removed the old lv for
> extra disk space, any new vgscan during initrd assigned my new root lv
> to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
> doesn't like the name).  There was no longer an lvm minor device 3 to
> mount.

This looks like the same problem that I am having, although I haven't
removed any logical volumes!

I am not 100% sure how to find the device number, but common sense tells me
that it is the "Block #" at the bottom of the logical volume information, in
which case my root LV is Block # 58:1

> Once I saw the problem, I recue booted and chrooted to my new lvm, and
> manually changed the minor number before running lilo.

Please can you tell me how you changed this?

Cheers,

Ben

RE: [linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Steven Lembark]

-- Ben Holness <ben@bens-house.org.uk>

>> Upon the next reboot, the initrd ran fine... found and activated the
>> lv's... but the kernel said "no root partition" and hung.  I could
>> rescue boot the system from the SuSE CD -- everything was there... I
>> finally added a "vgdisplay -v" to the initrd's linuxrc, and
>> that showed me the problem.

We can add it under the general heading of why using
LVM for boot volumes doesn't work on i386 platforms.

Giving /usr its own mount point genrally levaes the
root voume small, static and perfectly happy on a
small partition at the top of your disk. Using a
static link for LVM and installing it in the root
leaves the whole setup quite a bit more dependable.

--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 800 762 1582

[linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Chris Worley]

Sorry for the late reply... I don't subscribe to the list.

You've got to change the device minor number associated with the lvm 
partition.  I.E. if you partition name is /dev/foo/system, then you've 
got to get it to have the same minor device number vgscan is going to 
give during initrd.

If you're booted from some sort of rescue system that has a different 
root, "mount" your lvm root file system, "cd" to the directory where 
mounted, "chroot ." in that directory, manually delete and recreate 
(using "mknod") the /dev/foo/system file, with the correct minor 
number (the one vgscan is going to assign when initrd is running), 
then re-run lilo.

Note that this assumes that you've got a different root file system 
than the one you want, and the one you want won't boot because you 
deleted a lv with a lower minor number than your root partition.

<rant>I still think this should be documented under lvremove</rant>

Chris

Ben Holness writes:

>> Upon the next reboot, the initrd ran fine... found and activated the
>> lv's... but the kernel said "no root partition" and hung.  I could
>> rescue boot the system from the SuSE CD -- everything was there... I
>> finally added a "vgdisplay -v" to the initrd's linuxrc, and
>> that showed me the problem.
>>
>> The file system it was trying to mount was, to lilo, the device "3a:03"
>> (major device number 58, minor 3), but, once I'd removed the old lv for
>> extra disk space, any new vgscan during initrd assigned my new root lv
>> to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
>> doesn't like the name).  There was no longer an lvm minor device 3 to
>> mount.
> 
> This looks like the same problem that I am having, although I haven't
> removed any logical volumes!
> 
> I am not 100% sure how to find the device number, but common sense tells me
> that it is the "Block #" at the bottom of the logical volume information, in
> which case my root LV is Block # 58:1
> 
>> Once I saw the problem, I recue booted and chrooted to my new lvm, and
>> manually changed the minor number before running lilo.
> 
> Please can you tell me how you changed this?