[linux-lvm] Recovering from a disaster...

[linux-lvm] Recovering from a disaster...

[Shaw, Marco]

Going for my RHCE in a few weeks...  Always wondered:

1. Are there any gotchas in a corrupted LVM volume?  For example, /root/vg is an LVM volume, and either the 1st superblock on /root/vg or maybe the underlying /dev/sda1 partition or even /dev/md0 (assuming /root/vg could be sitting on top of a software RAID device) is zero'ed out.  How does one go about fixing it?  Will a simple fsck here work?  But in the above case, is fsck run against md0 or sda1?

2. If one knows that /dev/md0 is in trouble (composed of /dev/sda1 and /dev/sdb1), but /dev/sda1 is fine, can one, upon boot, simply change LILO/GRUB from trying to mount root from /dev/md0 to /dev/sda1?  My guess is no, since the disk label is different.

A typical scnerario could be LVM sitting on top of MD, and have to recover from some kind of disaster.  I wonder if the RedHat 8.0 rescue option has all the modules/tools to repair such a disaster.

A co-worker just screwed up a system I suspect was setup with a combination of LVM and MD.  I may have too quickly tried mounting /dev/sda2 (root on on disk), and maybe should have tried mounting /dev/md0.  After trying to mount /dev/sda2 (with an raid autodetect type), I think mount complained about invalid options, so I proceeded to do an fsck on /dev/sda2, only to have mention something about the superblock (yes, I should write all the errors down), so I tried fsck on something like 32768,
and it started up... And ran... And ran.  Result aftewards: all /dev/sda2 contains now after an attempt to mount it is "lost+found"!  :(

Marco

Re: [linux-lvm] Recovering from a disaster...

[Steven Lembark]

> Going for my RHCE in a few weeks...  Always wondered:
>
> 1. Are there any gotchas in a corrupted LVM volume?  For example,
> /root/vg is an LVM volume, and either the 1st superblock on /root/vg or
> maybe the underlying /dev/sda1 partition or even /dev/md0 (assuming
> /root/vg could be sitting on top of a software RAID device) is zero'ed
> out.  How does one go about fixing it?  Will a simple fsck here work?
> But in the above case, is fsck run against md0 or sda1?

Simple fix:

	Until Intel-based systems have a usable bios don't put your
	root/boot volume on LVM. Since the BIOS will not be upgraded
	until at least three weeks after pigs floss I'd strongly
	suggest avoiding the pratice altogether.

The main reason is that any glitch in the LVM system will
leave the system unbootable. The root volume is small enough
that it does not require LVM for either performance or multi-
PV distribution. Mirroring it will be better done at the
hardware level to avoid kernel glitches frying the mirror.

> 2. If one knows that /dev/md0 is in trouble (composed of /dev/sda1 and
> /dev/sdb1), but /dev/sda1 is fine, can one, upon boot, simply change
> LILO/GRUB from trying to mount root from /dev/md0 to /dev/sda1?  My guess
> is no, since the disk label is different.

> A typical scnerario could be LVM sitting on top of MD, and have to
> recover from some kind of disaster.  I wonder if the RedHat 8.0 rescue
> option has all the modules/tools to repair such a disaster.

They don't support LVM out of the box, so this is unlikely.


--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 800 762 1582

RE: [linux-lvm] Recovering from a disaster...

[Shaw, Marco]

>> A typical scnerario could be LVM sitting on top of MD, and have to 
>> recover from some kind of disaster.  I wonder if the RedHat 8.0 rescue 
>> option has all the modules/tools to repair such a disaster.

> They don't support LVM out of the box, so this is unlikely.

Actually, RH now supports LVM as of 8.0.  An lsmod when in 'rescue mode' also shows lvm_mod.

The whole disaster with software RAID and LVM mixed in, isn't easy, but a friend and I figured it out yesterday.  Not knowing the system has a mix of both before troubleshooting could be (actually is) disastrous!

Marco

Re: [linux-lvm] Recovering from a disaster...

[Mats Wichmann]

 >Simple fix:
 >
 >	Until Intel-based systems have a usable bios don't put your
 >	root/boot volume on LVM. Since the BIOS will not be upgraded
 >	until at least three weeks after pigs floss I'd strongly
 >	suggest avoiding the pratice altogether.

Ahem, to fuss a bit, the BIOS is a lodestone that comes
with the *PC* architecture.  AMD-processor PC systems are
similarly afflicted; while some Intel-processor systems
(notably Itanium) are not.  Someday we'll even see PC-class
machines without a BIOS.  So the comment about the likelihood
of upgrading the BIOS is undoubtedly true.

Re: [linux-lvm] Recovering from a disaster...

[bscott]

On Fri, 6 Dec 2002, at 11:38am, mats@laplaza.org wrote:
>> Until Intel-based systems have a usable bios don't put your
>> root/boot volume on LVM. Since the BIOS will not be upgraded
>> until at least three weeks after pigs floss I'd strongly
>> suggest avoiding the pratice altogether.
> 
> Ahem, to fuss a bit, the BIOS is a lodestone that comes with the *PC*
> architecture.

  And to fuss a bit more, "the BIOS" is an outdated term.  Most systems do
not have a single "BIOS" anymore; they have multiple components, each with
their own "BIOS".  These various "BIOSes" are just firmware; they contain
what is needed to run the device and bootstrap the OS (plus a ton of legacy
code leftover from the days of MS-DOS and floppy-only systems).

  Server-class systems built on the x86-compatible "PC" design now have many
(but not all) of the firmware features previously only found in propriatary
systems from Sun, SGI, etc.

> Someday we'll even see PC-class machines without a BIOS.

  Perhaps I misunderstand, but how does one boot without a BIOS?

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: [linux-lvm] Recovering from a disaster...

[Steven Dake]

>  
>
>>Someday we'll even see PC-class machines without a BIOS.
>>    
>>
>
>  Perhaps I misunderstand, but how does one boot without a BIOS?
>  
>
You could boot a PC class machine using EFI (which is really just a BIOS 
on steroids).  The nice thing about EFI is it isn't 16 bit, it can be 
developed easily in C, and its much faster then all that legacy 16 bit 
expansion rom PCI bootstrapping technique that really sucks.

-steve

>  
>

Re: [linux-lvm] Recovering from a disaster...

[Anders Widman]

>> Ahem, to fuss a bit, the BIOS is a lodestone that comes with the *PC*
>> architecture.

>   And to fuss a bit more, "the BIOS" is an outdated term.  Most systems do
> not have a single "BIOS" anymore; they have multiple components, each with
> their own "BIOS".  These various "BIOSes" are just firmware; they contain
> what is needed to run the device and bootstrap the OS (plus a ton of legacy
> code leftover from the days of MS-DOS and floppy-only systems).

>> Someday we'll even see PC-class machines without a BIOS.

>   Perhaps I misunderstand, but how does one boot without a BIOS?

Well,  The only thing needed is firmware for various devices that have
the  capability  to  load the OS from some form of media - either over
LAN/Internet, disks, floppy, CD, ROM, etc etc..

Todays  BIOS  are  often around 256KB with lots of old legacy code for
DOS based systems. Most of it is also 16 bit code.

- Anders

Re: [linux-lvm] Recovering from a disaster...

[bscott]

On Sat, 7 Dec 2002, at 10:45am, andewid@tnonline.net wrote:
>>> Someday we'll even see PC-class machines without a BIOS.
> 
>> Perhaps I misunderstand, but how does one boot without a BIOS?
> 
> Well,  The only thing needed is firmware for various devices that have
> the  capability  to  load the OS from some form of media - either over
> LAN/Internet, disks, floppy, CD, ROM, etc etc..
> 
> Todays  BIOS  are  often around 256KB with lots of old legacy code for
> DOS based systems. Most of it is also 16 bit code.

  Ah.  So you're not really talking about no BIOS at all, just removing all
the old code that isn't used anymore anyway.

  I've often wondered how much of that stuff still works, anyway.  We've
occasionally had customers ask us to move some ancient, MS-DOS-based
application new hardware.  Sometimes it works, sometimes it doesn't.  It
wouldn't surprise me if software rot has caused whole sections of the old
BIOS code to stop working.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: [linux-lvm] Recovering from a disaster...

[Steven Lembark]

>   And to fuss a bit more, "the BIOS" is an outdated term.  Most systems do
> not have a single "BIOS" anymore; they have multiple components, each with
> their own "BIOS".  These various "BIOSes" are just firmware; they contain
> what is needed to run the device and bootstrap the OS (plus a ton of
> legacy code leftover from the days of MS-DOS and floppy-only systems).
>
>   Server-class systems built on the x86-compatible "PC" design now have
> many (but not all) of the firmware features previously only found in
> propriatary systems from Sun, SGI, etc.

Among the things they don't have is a sane boot command prompt
that allows cleaning up the system by bypassing LVM on startup
(e.g., "boot -lm /dev/blah" on HP PARISC systems). And, yes,
this is true of Wintel-look-alike systems also, AMD has not
done any better job of leaving you with a command line at boot
to handle nasty situations.

Net result is that if LVM gets zapped you have no way to access
the root volume without it if your root volume is in LVM if you
have a Wintel motherboard or a clone or a lookalike or a nearly
similarly functional item of the general motherboard type (think
that covers all of it).

At least if your root volume (80MB of generally non-changing
space) is on a partition then you can boot it if LVM -- or
software RAID -- has problems and fix it using native tools
on the root voulme. Obviously, if you install your LVM or
RAID application and supporting files into /usr or /opt then
having the root volume online won't help much -- you can always
shoot yourself in the foot at some point. But with the simple
step of having recovery files on the root volume, booting single
user is usually sufficient to fix or recover things.

In theory people will have rescue CD's or floppys with the
current versions of everything on them that they can use to
recover it all; and printouts of the volume group configs;
and drive slowly enough that we don't have speed-related
accidents. Right. So, given the number of people who do get
zapped by this, the simple expedient of keeping the root
volume on a simple partition seems like a really easy fix...

>> Someday we'll even see PC-class machines without a BIOS.
>
>   Perhaps I misunderstand, but how does one boot without a BIOS?

Hardwire the behavior to the level that even a ROM-ed BIOS
cannot modify it. Cheaper than dealing with copyrighted
code and leaves the machines even more braindead at boot.
You can do it with a MUX and a timer: iterate the bus wires
kicking them with a "boot me" sequence until something responds.
No configuration required: everything directly addressable
via the MOBO's bus will get kicked in turn; first one wins.
You won't be able to boot off of external cards since the
wires don't run there but the board mfr's are into cheap not
effective.



--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 800 762 1582

Re: [linux-lvm] Recovering from a disaster...

[Luca Berra]

Steven Lembark wrote:

> Among the things they don't have is a sane boot command prompt
> that allows cleaning up the system by bypassing LVM on startup
> (e.g., "boot -lm /dev/blah" on HP PARISC systems). And, yes,
> this is true of Wintel-look-alike systems also, AMD has not
> done any better job of leaving you with a command line at boot
> to handle nasty situations.

i won't say about sgi or sun, but HP PARISC system do not provide this 
in firmware.


i have already posted on this list about hp systems, i'll do again (from 
memory so forgive any imprecisions)

hp 9000 firmware is able of identifying most hardware and of booting 
from disk, cd, tape or lan (dhcp + proprietary load method)

on the boot media ther is a 2Mb area called LIF area that contains an 
archive with some utilities (you can peek at containts using 'lifls 
/dev/rdsk/cXtYdZ)
hp firmware loads and runs something called ISL contained in this archive.
- end of firmware work (mostly) -

ISL is a secondary loader it can load other utilities contained in the lif.
HPUX is one of these utilities: it is able to read data from an hfs 
formatted filesystem and sets up the kernel using this data.
it can pass parameters to the kernel.
the magic
hpux -lm (device path)/path/to/kernel
works by reading a file contained in the same place as the kernel called 
rootconf, rootconf contains in binary form the offset on disk and lenght 
of the root filesystem.

That's all, it can be implemented with grub and linux lvm NOW if we want 
to accept the limit of having a fixed size contiguous root filesystem.

we can do better than that even with the crippling pc bios and we should 
be able to disregarding the firmware we are running from (assuming it is 
at least able to boot from one of (disk/tape/cd/floppy/lan)

I am strongly annoyed at ppl that continuously state 'dont put root on 
lvm because it is unsafe'.

the only difference between code parsing a partition table and code 
parsing lvm metadata is complexity, if you screw the partition table
(or if you rm /stand/rootconf on hp-ux AND screw LVM boot area) you are 
in the same situation.

linux already has the means of making it extremely safe and much more 
extensible than what other systems have. and if it is designed to work 
on the most crippled firmware we have (pc bios) it will certainly work 
with better firmware.

just create an lvm on disk format that reserves the first megabytes for 
a secondary loader (we can even use a linux kernel and it's initramfs as 
a secondary loader with 2.6).

tell the secondary loader where the root filesystem is placed (in case 
you screw LVM) it should not be a very complex map and it can be 
replicated in more than one place if necessary.

oh, we probably have to dump pc partition table format for this (too 
sorry :)

L.