capturing oopses

capturing oopses

[Folkert van Heusden]

Hi,

My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
would like to report it. Now the system locks up hard so I can't copy
the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
take a picture of the display?


Folkert van Heusden

-- 
Try MultiTail! Multiple windows with logfiles, filtered with regular
expressions, colored output, etc. etc. www.vanheusden.com/multitail/
----------------------------------------------------------------------
Get your PGP/GPG key signed at www.biglumber.com!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: capturing oopses

[bert hubert]

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> would like to report it. Now the system locks up hard so I can't copy
> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> take a picture of the display?

Try the serial port- you can get a copy of the console on ttyS0.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: capturing oopses

[Brice Goglin]

bert hubert wrote:

>On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
>  
>
>>My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
>>would like to report it. Now the system locks up hard so I can't copy
>>the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
>>don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
>>take a picture of the display?
>>    
>>
>
>Try the serial port- you can get a copy of the console on ttyS0.
>
>  
>
netconsole works great too.

Brice

Re: capturing oopses

[Folkert van Heusden]

> > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > would like to report it. Now the system locks up hard so I can't copy
> > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > take a picture of the display?
> Try the serial port- you can get a copy of the console on ttyS0.

No serial ports free unfortunately.


Folkert van Heusden

-- 
Try MultiTail! Multiple windows with logfiles, filtered with regular
expressions, colored output, etc. etc. www.vanheusden.com/multitail/
----------------------------------------------------------------------
Get your PGP/GPG key signed at www.biglumber.com!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: capturing oopses

[bert hubert]

On Wed, Nov 23, 2005 at 11:55:24AM +0100, Folkert van Heusden wrote:
> > Try the serial port- you can get a copy of the console on ttyS0.
> 
> No serial ports free unfortunately.

Then I suggest you whip out your trusty pencil.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services

Re: capturing oopses

[Adrian Bunk]

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:

> Hi,

Hi Folkert,

> My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> would like to report it. Now the system locks up hard so I can't copy
> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> take a picture of the display?

yes, digicams have become a common tool for reporting Oops'es.

> Folkert van Heusden

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: capturing oopses

[Ville Herva]

On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> 
> > Hi,
> 
> Hi Folkert,
> 
> > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > would like to report it. Now the system locks up hard so I can't copy
> > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > take a picture of the display?
> 
> yes, digicams have become a common tool for reporting Oops'es.

Speaking of which, does anybody know a feasible (as in "not too much harder
than manually typing it in manually") way to OCR characters from vga text
mode screen captures - or even digican shots? 

The vga text mode captures are from a remote administration interface (such
as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
be doable. The digican shots on the other hand... Well at least it would
have hack value :).

(My personal opinion is that Linus' unwillingness to include anything like
kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat unfortunate.)


-- v -- 

v@iki.fi

Re: capturing oopses

[Bodo Eggert]

Ville Herva <vherva@vianova.fi> wrote:

> Speaking of which, does anybody know a feasible (as in "not too much harder
> than manually typing it in manually") way to OCR characters from vga text
> mode screen captures - or even digican shots?
> 
> The vga text mode captures are from a remote administration interface (such
> as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
> be doable.

If it's a 640 pixel width image and you've got the right font loaded, the
VGA function 0x08 will get you the character at the current cursor position.
It's also used by PRTSCR, so pressing it in a DOS viewer may dump the text
to LPT1. If you redirected it to a serial console, you might also catch it.
(I never tried).
-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

Re: capturing oopses

[Ville Herva]

On Sun, Nov 27, 2005 at 02:10:59AM +0100, you [Bodo Eggert] wrote:
> 
> If it's a 640 pixel width image and you've got the right font loaded, the
> VGA function 0x08 will get you the character at the current cursor position.
> It's also used by PRTSCR, so pressing it in a DOS viewer may dump the text
> to LPT1. If you redirected it to a serial console, you might also catch it.
> (I never tried).

I meant that I have the image as a PNG (whatever), captured from the display
of the remote control program (HP RILOE, VMWare GSX Console, VNC, whatever).
I'm aware that the VGA display memory holds the characters (in addition to
pixels) in text mode, but the actual machine may be dead at this point - OS
crash, hw failure, whatever. The oops output is on the screen - just not in
palatible text format.


-- v -- 

v@iki.fi

Re: capturing oopses

[Sander]

Ville Herva wrote (ao):
> I meant that I have the image as a PNG (whatever), captured from the display
> of the remote control program (HP RILOE, VMWare GSX Console, VNC, whatever).
> I'm aware that the VGA display memory holds the characters (in addition to
> pixels) in text mode, but the actual machine may be dead at this point - OS
> crash, hw failure, whatever. The oops output is on the screen - just not in
> palatible text format.

Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
in Debian.

I had little success with a digicam photo of lightgray text on
not-quite-white paper, even after trying to improve things with gimp,
but it might work if you have the oops on screen.

Then again, if you have it on screen, you might as well put it on the
web.

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: capturing oopses

[Ville Herva]

On Sun, Nov 27, 2005 at 02:55:41PM +0100, you [Sander] wrote:
> 
> Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
> in Debian.
> 
> I had little success with a digicam photo of lightgray text on
> not-quite-white paper, even after trying to improve things with gimp,
> but it might work if you have the oops on screen.

Seems promising, but the results are not that great. See

http://v.iki.fi/~vherva/tmp/shot.png 
and
http://v.iki.fi/~vherva/tmp/shot.txt

the gray level setting didn't help much.
 
> Then again, if you have it on screen, you might as well put it on the
> web.

Sometimes you need to ksymoops the dump (for older kernels) or paste some
parts to some program.

In theory, converting a vga text screen capture shouldn't be hard, but in
practise... Well, it's much more beneficial to me to learn to type
passably...


-- v -- 

v@iki.fi

Re: capturing oopses

[Ville Herva]

On Sun, Nov 27, 2005 at 04:44:57PM +0200, you [Ville Herva] wrote:
> On Sun, Nov 27, 2005 at 02:55:41PM +0100, you [Sander] wrote:
> > 
> > Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
> > in Debian.
> > 
> > I had little success with a digicam photo of lightgray text on
> > not-quite-white paper, even after trying to improve things with gimp,
> > but it might work if you have the oops on screen.
> 
> Seems promising, but the results are not that great. See
> 
> http://v.iki.fi/~vherva/tmp/shot.png 
> and
> http://v.iki.fi/~vherva/tmp/shot.txt
> 
> the gray level setting didn't help much.

Ok, if I invert the image (black on white), I get
  http://v.iki.fi/~vherva/tmp/shot2.txt
which is fairly usable.


-- v -- 

v@iki.fi

Re: capturing oopses

[Randy.Dunlap]

On Sat, 26 Nov 2005 21:33:58 +0200 Ville Herva wrote:

> On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> > On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> > 
> > > Hi,
> > 
> > Hi Folkert,
> > 
> > > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > > would like to report it. Now the system locks up hard so I can't copy
> > > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > > take a picture of the display?
> > 
> > yes, digicams have become a common tool for reporting Oops'es.
> 
> Speaking of which, does anybody know a feasible (as in "not too much harder
> than manually typing it in manually") way to OCR characters from vga text
> mode screen captures - or even digican shots? 
> 
> The vga text mode captures are from a remote administration interface (such
> as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
> be doable. The digican shots on the other hand... Well at least it would
> have hack value :).
> 
> (My personal opinion is that Linus' unwillingness to include anything like
> kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat unfortunate.)

BTW, status of that:  it needs a little work to be more reliable.
(It hangs sometimes when switching from protected to real mode.)
I'm hoping that some of the APIC/IOAPIC/PIC patches that are being
done for kdump will also help kmsgdump.  I'll be working more on it
in the next few weeks/months.

so yes, when it's working, it's very useful IMO.

---
~Randy

Re: capturing oopses

[Maneesh Soni]

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:

> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and

could you try kdump (Documentation/kdump/kdump.txt) to capture the crash 
dump. It is part of mainline kernel atleast for i386 architecture. It 
should work for 2.6.14 also.

Thanks
Maneesh

-- 
Maneesh Soni
Linux Technology Center, 
IBM India Software Labs,
Bangalore, India
email: maneesh@in.ibm.com
Phone: 91-80-25044990

Re: capturing oopses

[mgross]

On Sunday 27 November 2005 20:41, Randy.Dunlap wrote:
> On Sat, 26 Nov 2005 21:33:58 +0200 Ville Herva wrote:
> > On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> > > On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> > > > Hi,
> > >
> > > Hi Folkert,
> > >
> > > > My 2.6.14 system occasionally crashes; gives a kernel panic. Of
> > > > course I would like to report it. Now the system locks up hard so I
> > > > can't copy the stacktrace. The crash dump patches mentioned in
> > > > oops-tracing.txt all don't work for 2.6.14 it seems. So: what should
> > > > I do? Get my digicam and take a picture of the display?
> > >
> > > yes, digicams have become a common tool for reporting Oops'es.
> >
> > Speaking of which, does anybody know a feasible (as in "not too much
> > harder than manually typing it in manually") way to OCR characters from
> > vga text mode screen captures - or even digican shots?
> >
> > The vga text mode captures are from a remote administration interface
> > (such as HP RILOE or vmware gsx console) so they are pixel perfect and
> > OCR should be doable. The digican shots on the other hand... Well at
> > least it would have hack value :).
> >
> > (My personal opinion is that Linus' unwillingness to include anything
> > like kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat
> > unfortunate.)
>
> BTW, status of that:  it needs a little work to be more reliable.
> (It hangs sometimes when switching from protected to real mode.)
> I'm hoping that some of the APIC/IOAPIC/PIC patches that are being
> done for kdump will also help kmsgdump.  I'll be working more on it
> in the next few weeks/months.
>
> so yes, when it's working, it's very useful IMO.
>

You know some platforms that perserve the memory above some addresses across 
warm boots.  For such platforms, one could reserve a buffer in that area can 
copy the sys log buffer to it on panic along with a bit pattern that could be 
searched for upon the next boot.

Additionally some platforms have flash parts that could be used as a 
persistant store for such data in the same manner.  I've done this on an 
XScale platform using an ancient (2.4.10) kernel.  I've always thought it 
would be a handy thing for the newer kernels for PC's, if we could come up 
with a semi-platform independent way of identifying a few pages that would 
survive a warm boot.

--mgross

Re: capturing oopses

[Christopher Friesen]

mgross wrote:

> You know some platforms that perserve the memory above some addresses across 
> warm boots.  For such platforms, one could reserve a buffer in that area can 
> copy the sys log buffer to it on panic along with a bit pattern that could be 
> searched for upon the next boot.

I've worked on a few blades that had this.  Serial console wasn't going 
to be available in the field, so being able to just dump debug 
information to persistent memory was really nice.

Chris

Re: capturing oopses

[Ville Herva]

On Mon, Nov 28, 2005 at 08:20:02AM -0800, you [mgross] wrote:
> 
> You know some platforms that perserve the memory above some addresses across 
> warm boots.  For such platforms, one could reserve a buffer in that area can 
> copy the sys log buffer to it on panic along with a bit pattern that could be 
> searched for upon the next boot.

I think there was a patch to do this a couple of years ago. 
 
The only references I can find now are:

http://groups.google.com/groups?q=%22Utility+module+to+capture+OOPS+output+over+reboot%22&hl=en&selm=fa.fbd0l7v.14hau3n%40ifi.uio.no&rnum=1)
http://marc.theaimsgroup.com/?l=linux-kernel&m=93077831203565&w=2


-- v -- 

v@iki.fi