writing OOPS/panic info to nvram?

writing OOPS/panic info to nvram?

[Roy Sigurd Karlsbakk]

hi

I just read in the OS X.2 technote 
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that 
they're writing the panic dump to nvram.

Is it hard to implement this on Linux?

roy
-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: writing OOPS/panic info to nvram?

[Morten Helgesen]

Hey, Roy

On Wed, Sep 04, 2002 at 01:50:21PM +0200, Roy Sigurd Karlsbakk wrote:
> hi
> 
> I just read in the OS X.2 technote 

So did I :)

> (http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that 
> they're writing the panic dump to nvram.
> 
> Is it hard to implement this on Linux?

Doesn`t look like too much work. As far as I can see, from a quick
glance, we have everything we need at hand - just have to write a snippet
of code in traps.c. I`ll give it a whirl now.

> 
> roy
> -- 
> Roy Sigurd Karlsbakk, Datavaktmester
> ProntoTV AS - http://www.pronto.tv/
> Tel: +47 9801 3356
> 

== Morten

-- 

"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
admin@nextframe.net / 93445641
http://www.nextframe.net

Re: writing OOPS/panic info to nvram?

[Alan Cox]

On Wed, 2002-09-04 at 12:50, Roy Sigurd Karlsbakk wrote:
> hi
> 
> I just read in the OS X.2 technote 
> (http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that 
> they're writing the panic dump to nvram.
> 
> Is it hard to implement this on Linux?

Its been done years ago. However on a PC you basically have no free
nvram so its not terribly useful there.

Re: writing OOPS/panic info to nvram?

[Morten Helgesen]

On Wed, Sep 04, 2002 at 01:21:33PM +0100, Alan Cox wrote:
> On Wed, 2002-09-04 at 12:50, Roy Sigurd Karlsbakk wrote:
> > hi
> > 
> > I just read in the OS X.2 technote 
> > (http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) that 
> > they're writing the panic dump to nvram.
> > 
> > Is it hard to implement this on Linux?
> 
> Its been done years ago. However on a PC you basically have no free
> nvram so its not terribly useful there.

True - the 'normal' size on a PC is apparently something like 114 bytes ... 
I  guess we could use it for something useful ... but maybe not for
OOPSen/panics. 

I didn`t realize we only had 114 bytes to work with.

== Morten

-- 

"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
admin@nextframe.net / 93445641
http://www.nextframe.net

Re: writing OOPS/panic info to nvram?

[Alan Cox]

On Wed, 2002-09-04 at 13:31, Morten Helgesen wrote:
> True - the 'normal' size on a PC is apparently something like 114 bytes ... 
> I  guess we could use it for something useful ... but maybe not for
> OOPSen/panics. 
> 
> I didn`t realize we only had 114 bytes to work with.

We don't. They are all used by the BIOS

Re: writing OOPS/panic info to nvram?

[Morten Helgesen]

On Wed, Sep 04, 2002 at 01:49:12PM +0100, Alan Cox wrote:
> On Wed, 2002-09-04 at 13:31, Morten Helgesen wrote:
> > True - the 'normal' size on a PC is apparently something like 114 bytes ... 
> > I  guess we could use it for something useful ... but maybe not for
> > OOPSen/panics. 
> > 
> > I didn`t realize we only had 114 bytes to work with.
> 
> We don't. They are all used by the BIOS

That makes it even less useful. Oh well.

== Morten

-- 

"Livet er ikke for nybegynnere" - sitat fra en klok person.

mvh
Morten Helgesen 
UNIX System Administrator & C Developer 
Nextframe AS
admin@nextframe.net / 93445641
http://www.nextframe.net

Re: writing OOPS/panic info to nvram?

[Remco Post]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On woensdag, september 4, 2002, at 02:54 , Morten Helgesen wrote:

> On Wed, Sep 04, 2002 at 01:49:12PM +0100, Alan Cox wrote:
>> On Wed, 2002-09-04 at 13:31, Morten Helgesen wrote:
>>> True - the 'normal' size on a PC is apparently something like 114 
>>> bytes ...
>>> I  guess we could use it for something useful ... but maybe not for
>>> OOPSen/panics.
>>>
>>> I didn`t realize we only had 114 bytes to work with.
>>
>> We don't. They are all used by the BIOS
>
> That makes it even less useful. Oh well.
>

For PC style hardware it does. For other platforms, it's stil nice to be 
able to see the oops info on an unattended crash (all crashes? ;) Dump 
to nvram, dump to file after boot.... Other option is to crash-dump to 
swap... Question is, do you really want to do that?

- ---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam    http://www.sara.nl
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer 
industry
didn't even foresee that the century was going to end." -- Douglas Adams


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)

iD8DBQE9dhJkBIoCv9yTlOwRAgG3AJ0SFZCTxz01okoJjlD4QHqNEBJjuACgp6sI
YJloQg8tfaE+677XsT7sq7M=
=p/b2
-----END PGP SIGNATURE-----

Re: writing OOPS/panic info to nvram?

[J.A. Magallon]

On 2002.09.04 Remco Post wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>
>On woensdag, september 4, 2002, at 02:54 , Morten Helgesen wrote:
>
>> On Wed, Sep 04, 2002 at 01:49:12PM +0100, Alan Cox wrote:
>>> On Wed, 2002-09-04 at 13:31, Morten Helgesen wrote:
>>>> True - the 'normal' size on a PC is apparently something like 114 
>>>> bytes ...
>>>> I  guess we could use it for something useful ... but maybe not for
>>>> OOPSen/panics.
>>>>
>>>> I didn`t realize we only had 114 bytes to work with.
>>>
>>> We don't. They are all used by the BIOS
>>
>> That makes it even less useful. Oh well.
>>
>
>For PC style hardware it does. For other platforms, it's stil nice to be 
>able to see the oops info on an unattended crash (all crashes? ;) Dump 
>to nvram, dump to file after boot.... Other option is to crash-dump to 
>swap... Question is, do you really want to do that?
>

Instead of swap, let user specify a partition to raw dump there. If a user
wants crash dumps, he has to leave some small disk space free and give an
option like "dump=/dev/hda7".

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.0 (Cooker) for i586
Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))

Re: writing OOPS/panic info to nvram?

[Remco Post]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On woensdag, september 4, 2002, at 04:08 , J.A. Magallon wrote:

>
> On 2002.09.04 Remco Post wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>> On woensdag, september 4, 2002, at 02:54 , Morten Helgesen wrote:
>>
>>> On Wed, Sep 04, 2002 at 01:49:12PM +0100, Alan Cox wrote:
>>>> On Wed, 2002-09-04 at 13:31, Morten Helgesen wrote:
>>>>> True - the 'normal' size on a PC is apparently something like 114
>>>>> bytes ...
>>>>> I  guess we could use it for something useful ... but maybe not for
>>>>> OOPSen/panics.
>>>>>
>>>>> I didn`t realize we only had 114 bytes to work with.
>>>>
>>>> We don't. They are all used by the BIOS
>>>
>>> That makes it even less useful. Oh well.
>>>
>>
>> For PC style hardware it does. For other platforms, it's stil nice to 
>> be
>> able to see the oops info on an unattended crash (all crashes? ;) Dump
>> to nvram, dump to file after boot.... Other option is to crash-dump to
>> swap... Question is, do you really want to do that?
>>
>
> Instead of swap, let user specify a partition to raw dump there. If a 
> user
> wants crash dumps, he has to leave some small disk space free and give 
> an
> option like "dump=/dev/hda7".
>
>

Mjah,

swap can be destroyed during a crash dump, so why not use that? 
Everybody else does it (AIX, Solaris, IRIX) so if none of the commercial 
UNIX vendors see a problem in using swap for dump, why should we?

I like the nvram option best for hardware that allows a oops output to 
be stored in nvram, since that is also usable on diskless systems when 
the network is down (or the network driver oopses) or when the disk 
driver oopses... Maybe do something like:

if there is enough space on disk && ..., use that else
if there is a swap over nfs && ..., use that else
if there is a tape drive attaced and a tape is present and it is 
writeable... else
if there is nvram available use that

and print on the console....

maybe this is all a bit overkill for what we need to do, maybe not... I 
guess it would make it possible to debug systems that are just sitting 
there by themselves, and have some curious problem...

- ---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam    http://www.sara.nl
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer 
industry
didn't even foresee that the century was going to end." -- Douglas Adams


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (Darwin)

iD8DBQE9dhdQBIoCv9yTlOwRAsm0AJ40VJx5QmhrI7ME4w02WyM3Vn9X6wCcDMyq
M8Dp4XhDjSkKtEO3rbLlwxo=
=YtBZ
-----END PGP SIGNATURE-----

Re: writing OOPS/panic info to nvram?

[Pavel Machek]

Hi!

> driver oopses... Maybe do something like:
> 
> if there is enough space on disk && ..., use that else
> if there is a swap over nfs && ..., use that else
> if there is a tape drive attaced and a tape is present and it is 
> writeable... else
> if there is nvram available use that

You just killed any data you had on the tape... too bad.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: writing OOPS/panic info to nvram?

[Remco Post]

On vrijdag, september 6, 2002, at 12:06 , Pavel Machek wrote:

> Hi!
>
>> driver oopses... Maybe do something like:
>>
>> if there is enough space on disk && ..., use that else
>> if there is a swap over nfs && ..., use that else
>> if there is a tape drive attaced and a tape is present and it is
>> writeable... else
>> if there is nvram available use that
>
> You just killed any data you had on the tape... too bad.
> 								Pavel

Yes, so, or you just saved that oops that has been bugging you for 
months... (And yes I'm probably one of those rare people that has 
tapedrives attached that are not used for anything usefull).

---
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam    http://www.sara.nl
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167
PGP keys at http://home.sara.nl/~remco/keys.asc

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer 
industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: writing OOPS/panic info to nvram?

[Pavel Machek]

Hi!

> >>driver oopses... Maybe do something like:
> >>
> >>if there is enough space on disk && ..., use that else
> >>if there is a swap over nfs && ..., use that else
> >>if there is a tape drive attaced and a tape is present and it is
> >>writeable... else
> >>if there is nvram available use that
> >
> >You just killed any data you had on the tape... too bad.
> >								Pavel
> 
> Yes, so, or you just saved that oops that has been bugging you for 
> months... (And yes I'm probably one of those rare people that has 
> tapedrives attached that are not used for anything usefull).

If it was bugging you for months, then you'd have probably copied it
using paper and pencil already. cli(); while(1); at the end of oops
handler is not *that* hard to do.
								Pavel
-- 
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?

Re: writing OOPS/panic info to nvram?

[Alan Cox]

On Wed, 2002-09-04 at 15:08, J.A. Magallon wrote:
> Instead of swap, let user specify a partition to raw dump there. If a user
> wants crash dumps, he has to leave some small disk space free and give an
> option like "dump=/dev/hda7".

With what will you write it - not the linux block layer thats for sure.
Ingo has patches for doing network dumps which are kind of neat

Re: writing OOPS/panic info to nvram?

[J.A. Magallon]

On 2002.09.04 Alan Cox wrote:
>On Wed, 2002-09-04 at 15:08, J.A. Magallon wrote:
>> Instead of swap, let user specify a partition to raw dump there. If a user
>> wants crash dumps, he has to leave some small disk space free and give an
>> option like "dump=/dev/hda7".
>
>With what will you write it - not the linux block layer thats for sure.
>Ingo has patches for doing network dumps which are kind of neat
>

Ah, ther is no way to write raw blocks at a very low level to disk...??
LKCD at least writes to a floppy, doesn't it.?
Say you just need one block for the dump. Could you get the block location
(H/C/S) on boot and tell the bios to write a chunk there on crash ?

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.0 (Cooker) for i586
Linux 2.4.20-pre5-j0 (gcc 3.2 (Mandrake Linux 9.0 3.2-1mdk))

Re: writing OOPS/panic info to nvram?

[Lars Marowsky-Bree]

On 2002-09-04T16:41:54,
   "J.A. Magallon" <jamagallon@able.es> said:

> Ah, ther is no way to write raw blocks at a very low level to disk...??

Not reliably; you _know_ your infrastructure has crashed, otherwise you
wouldn't be inside the crash dump handler ;), so you can't possibly trust the
normal block layer to write the crash dump (and not write it over your salary
and customer database).

Parts of this could probably be circumvented by a checksum of the code which
is computed at boot time and checked before the crashdump takes place, but this doesn't deal with a crashed SCSI driver.

A network dump is much safer, though I would suggest running it over a
dedicated card / driver combo and on a special ethernet protocol, because you
might have lost your IP configuration...


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Immortality is an adequate definition of high availability for me.
	--- Gregory F. Pfister

Re: writing OOPS/panic info to nvram?

[Pavel Machek]

Hi!

> > Ah, ther is no way to write raw blocks at a very low level to disk...??
> 
> Not reliably; you _know_ your infrastructure has crashed, otherwise you
> wouldn't be inside the crash dump handler ;), so you can't possibly trust the
> normal block layer to write the crash dump (and not write it over your salary
> and customer database).

Floppy seems like safe choice. Verify its special "crash floppy" by checking
signature, then write.

> A network dump is much safer, though I would suggest running it over a
> dedicated card / driver combo and on a special ethernet protocol, because you
> might have lost your IP configuration...

Its enough for it to work 99% cases. Separate ethernet card is overkill,
serial console is easier than that.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: writing OOPS/panic info to nvram?

[Paul Mackerras]

Alan Cox writes:

> With what will you write it - not the linux block layer thats for sure.
> Ingo has patches for doing network dumps which are kind of neat

Rusty has written a very basic polled-mode IDE driver for precisely
this situation.  He even tested it on x86 and powermac. :)  He has a
little user-space program that allocates a file and uses FIOBMAP to
work out which disk blocks it is using.  The program writes a
signature to the blocks and then tells the kernel crashdump module the
block numbers.  When the kernel panics, it calls the crashdump module
which first reads the blocks it was told and makes sure they have the
right signature, then writes the oops information to those blocks and
then reboots.

IDE was relatively straightforward since you can do basic block I/O
with just the ATA-1 or ATA-2 registers and command set and PIO.  In
contrast, I believe SCSI defeated him. :)

Paul.

Re: writing OOPS/panic info to nvram?

[David S. Miller]

   From: Paul Mackerras <paulus@au1.ibm.com>
   Date: Thu, 5 Sep 2002 09:34:52 +1000 (EST)
   
   I believe SCSI defeated him. :)

If you can get at the interrupt handler, and the interrupt handler is
coded well enough to handle sharing interrupts (%99 of PCI scsi
drivers are) then it is doable.  Just submit it as normal through
scsi, mark it high priority somehow, and then keep calling the
interrupt handler in a loop until your command completes or fails :-)

Re: writing OOPS/panic info to nvram?

[Alan Cox]

On Thu, 2002-09-05 at 00:34, Paul Mackerras wrote:
> IDE was relatively straightforward since you can do basic block I/O
> with just the ATA-1 or ATA-2 registers and command set and PIO.  In
> contrast, I believe SCSI defeated him. :)

You have to reset and retune the interface/controller registers as well,
otherwise bad things can happen. Rusty will be happy to know that next
generation SATA drops the PIO interface for a demented PIO via DMA setup
8)

Re: writing OOPS/panic info to nvram?

[Rusty Russell]

On 05 Sep 2002 01:07:01 +0100
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> On Thu, 2002-09-05 at 00:34, Paul Mackerras wrote:
> > IDE was relatively straightforward since you can do basic block I/O
> > with just the ATA-1 or ATA-2 registers and command set and PIO.  In
> > contrast, I believe SCSI defeated him. :)
> 
> You have to reset and retune the interface/controller registers as well,
> otherwise bad things can happen.

No, at this stage the code reboots the machine (well, I could call the ide
reset code at this point I guess).

Note that my mini oops dumper object file is a leafnode which doesn't use any
external code in the dump path (checked at build time).  It is armed with the
symbols and device & block offsets by userspace.  I plan to update it in the
next month, but it's trivial enough (a new driver with one hook in the oops
code) to be done after the freeze if reqd.

The interesting bit becomes harvesting those reports: this is a higher level
problem (userspace and privacy being the higher levels, respectively).

Rusty.
-- 
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy

Re: writing OOPS/panic info to nvram?

[Suparna Bhattacharya]

On Wed, 04 Sep 2002 20:02:40 +0530, Alan Cox wrote:

> On Wed, 2002-09-04 at 15:08, J.A. Magallon wrote:
>> Instead of swap, let user specify a partition to raw dump there. If a
>> user wants crash dumps, he has to leave some small disk space free and
>> give an option like "dump=/dev/hda7".
> 
> With what will you write it - not the linux block layer thats for sure.
> Ingo has patches for doing network dumps which are kind of neat
> 
> -


LKCD for 2.5 (WIP) has a dump driver interface through which different 
target types can be plugged in. For example Ingo's polled network dump 
code been integrated as one such dump driver target (generic type),
block layer based i/o is available as another target (for those
who chose to use it for their raw partition).Down the line specific
dump drivers suited for the hardware concerned, e.g Rusty's polled
IDE driver, could be plugged in as dump target too, as could NECs
work on converting dump block i/o to polled mode. 

Conceivably, one may perhaps have alternate targets available on the 
same system and failover to the suitable one based on the situation.
(If the network interface code is the one in trouble, try to
dump to the dedicated raw disk or vice versa).

And then, a little later there could be the option of memory save 
option abstracted as another driver target, to be followed 
by a soft-reboot (w/o clearing memory) for performing actual dump i/o
to persistent storage on architectures where this option works out.

Regards
Suparna

Re: writing OOPS/panic info to nvram?

[Christoph Stueckjuergen]

>On Wed, 2002-09-04 at 12:50, Roy Sigurd Karlsbakk wrote: 
> > hi 
> > 
> > I just read in the OS X.2 technote 
> >  
(http://developer.apple.com/technotes/tn2002/tn2053.html#TN001016) 
> > that
> > they're writing the panic dump to nvram. 
> > 
> > Is it hard to implement this on Linux? 

 > Its been done years ago. However on a PC you basically have no 
free 
 > nvram so its not terribly useful there. 
If it has been done years ago, are there patches also for 2.6 
kernels available?

Christoph