Re: oops pauser.

Re: oops pauser.

[Robert Hancock]

Dave Jones wrote:
> After an oops, we can't really rely on anything. What if the
> oops came from the console layer, or a framebuffer driver?

In this case, maybe it wouldn't work.. but I think it would be helpful a 
big majority of the time.

Surely there must be a way this could be done. After all, Windows seems 
to manage to fairly reliably switch the display into VGA 80x50 mode and 
put up the BSOD with error dump information with reasonably good 
reliability..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: oops pauser.

[Jan Engelhardt]

>> After an oops, we can't really rely on anything. What if the
>> oops came from the console layer, or a framebuffer driver?

How about this?:

Put an "emergency kernel" into a memory location that is being protected in 
some way (i.e. writing there even from kernel space generates an oops). 
Upon oops, it gets unlocked and we do some sort of kexec() to it.
Of course, this probably requires that the unlocking must not be done 
with help of the standard page mappings.


Jan Engelhardt
-- 

Re: oops pauser.

[Randy.Dunlap]

On Fri, 6 Jan 2006 08:06:12 +0100 (MET) Jan Engelhardt wrote:

> >> After an oops, we can't really rely on anything. What if the
> >> oops came from the console layer, or a framebuffer driver?
> 
> How about this?:
> 
> Put an "emergency kernel" into a memory location that is being protected in 
> some way (i.e. writing there even from kernel space generates an oops). 
> Upon oops, it gets unlocked and we do some sort of kexec() to it.
> Of course, this probably requires that the unlocking must not be done 
> with help of the standard page mappings.

This is what kexec + kdump is.

---
~Randy

Re: oops pauser.

[Dave Jones]

On Fri, Jan 06, 2006 at 08:06:12AM +0100, Jan Engelhardt wrote:
 > >> After an oops, we can't really rely on anything. What if the
 > >> oops came from the console layer, or a framebuffer driver?
 > 
 > How about this?:
 > 
 > Put an "emergency kernel" into a memory location that is being protected in 
 > some way (i.e. writing there even from kernel space generates an oops). 
 > Upon oops, it gets unlocked and we do some sort of kexec() to it.

You just reinvented 'kdump' :)
There's been ongoing work in this area for a while.

		Dave

Re: Console debugging wishlist was: Re: oops pauser.

[Bodo Eggert]

Andi Kleen <ak@suse.de> wrote:

> (it is hard to understand that with 128MB+ graphic cards and 512+MB
> computers the scroll back must be still so short...)

The VGA scrollback buffer is limited by the text area of the video RAM.
The text area is in the DOS memory at 0xB800 (or 0xB000) and extends
32 KB (or in case of MDA, 4 KB). Each character will use 2 Bytes.
Therefore you can store up to 16,000 characters or 4 pages of text.

-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Sunday 15 January 2006 17:48, Bodo Eggert wrote:
> Andi Kleen <ak@suse.de> wrote:
> 
> > (it is hard to understand that with 128MB+ graphic cards and 512+MB
> > computers the scroll back must be still so short...)
> 
> The VGA scrollback buffer is limited by the text area of the video RAM.
> The text area is in the DOS memory at 0xB800 (or 0xB000) and extends
> 32 KB (or in case of MDA, 4 KB). Each character will use 2 Bytes.
> Therefore you can store up to 16,000 characters or 4 pages of text.

It was a rhetorical question.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>> > (it is hard to understand that with 128MB+ graphic cards and 512+MB
>> > computers the scroll back must be still so short...)
>> 
>> The VGA scrollback buffer is limited by the text area of the video RAM.
>> The text area is in the DOS memory at 0xB800 (or 0xB000) and extends
>> 32 KB (or in case of MDA, 4 KB). Each character will use 2 Bytes.
>> Therefore you can store up to 16,000 characters or 4 pages of text.
>
>It was a rhetorical question.
>
And I assumed that scrollback was stored in some regular kmalloc()ed page(s).


Jan Engelhardt
-- 

oops pauser.

[Dave Jones]

In my quest to get better debug data from users in Fedora bug reports,
I came up with this patch.  A majority of users don't have serial
consoles, so when an oops scrolls off the top of the screen,
and locks up, they usually end up reporting a 2nd (or later) oops
that isn't particularly helpful (or worse, some inconsequential
info like 'sleeping whilst atomic' warnings)

With this patch, if we oops, there's a pause for a two minutes..
which hopefully gives people enough time to grab a digital camera
to take a screenshot of the oops.

It has an on-screen timer so the user knows what's going on,
(and that it's going to come back to life [maybe] after the oops).

The one case this doesn't catch is the problem of oopses whilst
in X. Previously a non-fatal oops would stall X momentarily,
and then things continue. Now those cases will lock up completely
for two minutes. Future patches could add some additional feedback
during this 'stall' such as the blinky keyboard leds, or periodic speaker beeps.

Signed-off-by: Dave Jones <davej@redhat.com>

--- vanilla/arch/i386/kernel/traps.c	2006-01-02 22:21:10.000000000 -0500
+++ linux-2.6.15/arch/i386/kernel/traps.c	2006-01-04 23:42:46.000000000 -0500
@@ -256,6 +271,15 @@ void show_registers(struct pt_regs *regs
 		}
 	}
 	printk("\n");
+	{
+		int i;
+		for (i=120;i>0;i--) {
+			mdelay(1000);
+			touch_nmi_watchdog();
+			printk("Continuing in %d seconds. \r", i);
+		}
+		printk("\n");
+	}
 }	
 
 static void handle_BUG(struct pt_regs *regs)

Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

Dave Jones <davej@redhat.com> writes:

> In my quest to get better debug data from users in Fedora bug reports,
> I came up with this patch.  A majority of users don't have serial
> consoles, so when an oops scrolls off the top of the screen,
> and locks up, they usually end up reporting a 2nd (or later) oops
> that isn't particularly helpful (or worse, some inconsequential
> info like 'sleeping whilst atomic' warnings)

Ok - here's my personal wishlist. If someone is interested ...

What I would like to have is a "more" option for the kernel that makes
it page kernel output like "more" and asks you before scrolling
to the next page.

What would be also cool would be to fix the VGA console to have 
a larger scroll back buffer.  The standard kernel boot output 
is far larger than the default scrollback, so if you get a hang
late you have no way to look back to all the earlier 
messages.

(it is hard to understand that with 128MB+ graphic cards and 512+MB
computers the scroll back must be still so short...) 

And fixing sysrq to work after panics would be also nice.

And maybe a sysrq key to switch the font to the smallest one available
so as much as possible would fit onto a digital photo.
> 
> The one case this doesn't catch is the problem of oopses whilst
> in X. Previously a non-fatal oops would stall X momentarily,
> and then things continue. Now those cases will lock up completely
> for two minutes. Future patches could add some additional feedback
> during this 'stall' such as the blinky keyboard leds, or periodic speaker beeps.

That's the killer issues why this patch is a bad idea.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>Ok - here's my personal wishlist. If someone is interested ...
>
>What I would like to have is a "more" option for the kernel that makes
>it page kernel output like "more" and asks you before scrolling
>to the next page.

An oops is usually a condition you can recover from in some/most/depends 
cases (e.g. a null deref in a filesystem "only" makes that vfsmount 
(filesystem at all?) blocked), so if the kernel is waiting for user input 
on a non-panic condition, this means userspace stops too, which is not 
too good if the kernel is still 'alive'.
It's like we are entering kdb although everything is fine enough to go 
through a proper `init 6`.

>What would be also cool would be to fix the VGA console to have 
>a larger scroll back buffer.  The standard kernel boot output 
>is far larger than the default scrollback, so if you get a hang
>late you have no way to look back to all the earlier 
>messages.
>
>(it is hard to understand that with 128MB+ graphic cards and 512+MB
>computers the scroll back must be still so short...) 

I doubt this scrollback buffer is implemented as part of the video cards. 
It is rather a kernel invention, and therefore uses standard RAM. But the 
idea is good, preferably make it a CONFIG_ option.

>And fixing sysrq to work after panics would be also nice.

I am not sure, but would enabling interrupts be enough?

>And maybe a sysrq key to switch the font to the smallest one available
>so as much as possible would fit onto a digital photo.

And analog photos? ;)

>> The one case this doesn't catch is the problem of oopses whilst
>> in X. Previously a non-fatal oops would stall X momentarily,
>> and then things continue. Now those cases will lock up completely
>> for two minutes. Future patches could add some additional feedback
>> during this 'stall' such as the blinky keyboard leds, or periodic speaker beeps.
>
>That's the killer issues why this patch is a bad idea.
>

Whilst few can be done in X situations, let's at least improve consoles.


Jan Engelhardt
-- 

Re: Console debugging wishlist was: Re: oops pauser.

[Josef Sipek]

On Tue, Jan 10, 2006 at 09:25:46PM +0100, Jan Engelhardt wrote:
> >What would be also cool would be to fix the VGA console to have 
> >a larger scroll back buffer.  The standard kernel boot output 
> >is far larger than the default scrollback, so if you get a hang
> >late you have no way to look back to all the earlier 
> >messages.
> >
> >(it is hard to understand that with 128MB+ graphic cards and 512+MB
> >computers the scroll back must be still so short...) 
> 
> I doubt this scrollback buffer is implemented as part of the video cards. 
> It is rather a kernel invention, and therefore uses standard RAM. But the 
> idea is good, preferably make it a CONFIG_ option.

There is a config option that lets you specify the size of this buffer:
CONFIG_LOG_BUF_SHIFT

Jeff.

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>> I doubt this scrollback buffer is implemented as part of the video cards. 
>> It is rather a kernel invention, and therefore uses standard RAM. But the 
>> idea is good, preferably make it a CONFIG_ option.
>
>There is a config option that lets you specify the size of this buffer:
>CONFIG_LOG_BUF_SHIFT

menuconfig help says

    "Select kernel log buffer size as a power of 2."

That does not sound like "console scroll buffer".



Jan Engelhardt
-- 

Re: Console debugging wishlist was: Re: oops pauser.

[Josef Sipek]

On Tue, Jan 10, 2006 at 09:44:43PM +0100, Jan Engelhardt wrote:
> 
> >> I doubt this scrollback buffer is implemented as part of the video cards. 
> >> It is rather a kernel invention, and therefore uses standard RAM. But the 
> >> idea is good, preferably make it a CONFIG_ option.
> >
> >There is a config option that lets you specify the size of this buffer:
> >CONFIG_LOG_BUF_SHIFT
> 
> menuconfig help says
> 
>     "Select kernel log buffer size as a power of 2."
> 
> That does not sound like "console scroll buffer".

True. I should think more about what I say before I say it.

Jeff.

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Tuesday 10 January 2006 21:29, Josef Sipek wrote:

> There is a config option that lets you specify the size of this buffer:
> CONFIG_LOG_BUF_SHIFT

That is the dmesg buffer, not the scroll back buffer. Completely different
things.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Tuesday 10 January 2006 21:25, Jan Engelhardt wrote:
> An oops is usually a condition you can recover from in some/most/depends 
> cases (e.g. a null deref in a filesystem "only" makes that vfsmount 
> (filesystem at all?) blocked), so if the kernel is waiting for user input 
> on a non-panic condition, this means userspace stops too, which is not 
> too good if the kernel is still 'alive'.
> It's like we are entering kdb although everything is fine enough to go 
> through a proper `init 6`.

-ENOPARSE

> 
> >What would be also cool would be to fix the VGA console to have 
> >a larger scroll back buffer.  The standard kernel boot output 
> >is far larger than the default scrollback, so if you get a hang
> >late you have no way to look back to all the earlier 
> >messages.
> >
> >(it is hard to understand that with 128MB+ graphic cards and 512+MB
> >computers the scroll back must be still so short...) 
> 
> I doubt this scrollback buffer is implemented as part of the video cards. 
> It is rather a kernel invention, and therefore uses standard RAM. But the 
> idea is good, preferably make it a CONFIG_ option.

At least long ago (when I last looked) it was in video RAM. 

> 
> >And fixing sysrq to work after panics would be also nice.
> 
> I am not sure, but would enabling interrupts be enough?

Interrupts are already enabled, but no - it's not.

Thank you for an useful contribution to the thread.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>-ENOPARSE

Try the oops.ko from http://jengelh.hopto.org/f/oops_ko.tbz2. It won't kill
your system, you can continue to work.

If you now had a kernel-level pager that would jump in everytime an oops
happened, control would normally not be given back to userspace unless we quit
the pager. kdb has a similar behavior: it "stops" userspace until someone
chooses to "c"ontinue.
Therefore this pager would not be too good. In a panic, yes, it would be 
perfect.

I hope this makes it a little bit clearer, if not, -EAGAIN.

>> >(it is hard to understand that with 128MB+ graphic cards and 512+MB
>> >computers the scroll back must be still so short...) 
>> 
>> I doubt this scrollback buffer is implemented as part of the video cards. 
>> It is rather a kernel invention, and therefore uses standard RAM. But the 
>> idea is good, preferably make it a CONFIG_ option.
>
>At least long ago (when I last looked) it was in video RAM. 

Let's put it from another POV: if the scrollback buffer was somewhere within
the video card, it would usually not be cleared when you change from one
console tty to another. Currently, doing this switch clears the buffer (can we
do anything about that? - would be great)



Jan Engelhardt
-- 

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Tuesday 10 January 2006 22:06, Jan Engelhardt wrote:

> If you now had a kernel-level pager that would jump in everytime an oops
> happened, control would normally not be given back to userspace unless we quit
> the pager. kdb has a similar behavior: it "stops" userspace until someone
> chooses to "c"ontinue.
> Therefore this pager would not be too good. In a panic, yes, it would be 
> perfect.

First for an recoverable oops there is no reason you couldn't use
schedule_timeout(). And for those you don't need it anyways
because you can as well use dmesg. For others you can use poll loops.

But it wasn't actually my point. If you get 
an problem during bootup - not necessarily an oops, but could
be also a no root panic or your SCSI controller not working or 
something else - and you can reproduce it it's a PITA to examine
the kernel output before because there is no way to get
enough scrollback.  For the oops itself it's not needed - it typically
fits on the screen. But if it happens every boot it would be nice
if you could just boot with "more" and then page through
the kernel output and check what's going on.

The feature would be mainly useful for problems during kernel bootup,
although it might be sometimes useful too e.g. when user space
hangs, but you want to page through the hotkey process dump
which might be longer than console scrollback.

Just more scrollback does not necessarily replace this because
sometimes youe end up with so much output so quickly (e.g. some errors
are very verbose) that any scrollback buffer would be overflown.

Now the only issue would be to work out when to use schedule_timeout
and when to use a delay, but that can be all distingushed with some code.

Anyways mind you - i suspect actually implementing this would be somewhat
ugly, so the chances of it actually getting in would be likely slim.
Still it would be often useful.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>But it wasn't actually my point. If you get 
>an problem during bootup - not necessarily an oops, but could
>be also a no root panic or your SCSI controller not working or 
>something else - and you can reproduce it it's a PITA to examine
>the kernel output before because there is no way to get
>enough scrollback.  For the oops itself it's not needed - it typically
>fits on the screen. But if it happens every boot it would be nice
>if you could just boot with "more" and then page through
>the kernel output and check what's going on.

Ah yes, I had not considered boot oopses/panics. My bad.



Jan Engelhardt
-- 

Re: Console debugging wishlist was: Re: oops pauser.

[Antonino A. Daplas]

Jan Engelhardt wrote:
>> Ok - here's my personal wishlist. If someone is interested ...
>>
>> What I would like to have is a "more" option for the kernel that makes
>> it page kernel output like "more" and asks you before scrolling
>> to the next page.
> 
> An oops is usually a condition you can recover from in some/most/depends 
> cases (e.g. a null deref in a filesystem "only" makes that vfsmount 
> (filesystem at all?) blocked), so if the kernel is waiting for user input 
> on a non-panic condition, this means userspace stops too, which is not 
> too good if the kernel is still 'alive'.
> It's like we are entering kdb although everything is fine enough to go 
> through a proper `init 6`.
> 
>> What would be also cool would be to fix the VGA console to have 
>> a larger scroll back buffer.  The standard kernel boot output 
>> is far larger than the default scrollback, so if you get a hang
>> late you have no way to look back to all the earlier 
>> messages.
>>
>> (it is hard to understand that with 128MB+ graphic cards and 512+MB
>> computers the scroll back must be still so short...) 
> 
> I doubt this scrollback buffer is implemented as part of the video cards. 
> It is rather a kernel invention, and therefore uses standard RAM. But the 
> idea is good, preferably make it a CONFIG_ option.

In the VGA console, all buffers, including scrollback is in video RAM, but
the size is fixed and is very small.

With the framebuffer console, you can increase the size of the scrollback
buffer with the boot option:

fbcon=scrollback:<n> (default is 32K)

Tony

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Wednesday 11 January 2006 13:24, Antonino A. Daplas wrote:

> In the VGA console, all buffers, including scrollback is in video RAM, but
> the size is fixed and is very small.

I wonder if that can be fixed.

> With the framebuffer console, you can increase the size of the scrollback
> buffer with the boot option:
> 
> fbcon=scrollback:<n> (default is 32K)

On x86-64 vesafb is unusable slow because it does CPU scrolling cause
it can't use the vesa BIOS - and the others don't work everywhere. So I don't
think fbcon is an usable replacement.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Antonino A. Daplas]

Andi Kleen wrote:
> On Wednesday 11 January 2006 13:24, Antonino A. Daplas wrote:
> 
>> In the VGA console, all buffers, including scrollback is in video RAM, but
>> the size is fixed and is very small.
> 
> I wonder if that can be fixed.

It can be done, but it will affect VGA console performance.
 
> 
>> With the framebuffer console, you can increase the size of the scrollback
>> buffer with the boot option:
>>
>> fbcon=scrollback:<n> (default is 32K)
> 
> On x86-64 vesafb is unusable slow because it does CPU scrolling cause
> it can't use the vesa BIOS - and the others don't work everywhere. So I don't
> think fbcon is an usable replacement.

How about vga16fb + fbcon? If scrolling is slow in vga16fb, fbset -vyres 800 should
increase performance significantly.

Tony

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Wednesday 11 January 2006 14:05, Antonino A. Daplas wrote:
> Andi Kleen wrote:
> > On Wednesday 11 January 2006 13:24, Antonino A. Daplas wrote:
> > 
> >> In the VGA console, all buffers, including scrollback is in video RAM, but
> >> the size is fixed and is very small.
> > 
> > I wonder if that can be fixed.
> 
> It can be done, but it will affect VGA console performance.

By how much? As long as it still scrolls reasonably fast it would be ok for me.

>  
> > 
> >> With the framebuffer console, you can increase the size of the scrollback
> >> buffer with the boot option:
> >>
> >> fbcon=scrollback:<n> (default is 32K)
> > 
> > On x86-64 vesafb is unusable slow because it does CPU scrolling cause
> > it can't use the vesa BIOS - and the others don't work everywhere. So I don't
> > think fbcon is an usable replacement.
> 
> How about vga16fb + fbcon? If scrolling is slow in vga16fb, fbset -vyres 800 should
> increase performance significantly.

I can try it.

-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Antonino A. Daplas]

Andi Kleen wrote:
> On Wednesday 11 January 2006 14:05, Antonino A. Daplas wrote:
>> Andi Kleen wrote:
>>> On Wednesday 11 January 2006 13:24, Antonino A. Daplas wrote:
>>>
>>>> In the VGA console, all buffers, including scrollback is in video RAM, but
>>>> the size is fixed and is very small.
>>> I wonder if that can be fixed.
>> It can be done, but it will affect VGA console performance.
> 
> By how much? As long as it still scrolls reasonably fast it would be ok for me.

Each character will need to be written twice, one to VGA RAM and another to
the shadow/scrollback buffer in system RAM. It would still be reasonably fast.

Perhaps I can implement this for vgacon.

Tony

Re: Console debugging wishlist was: Re: oops pauser.

[Andi Kleen]

On Wednesday 11 January 2006 14:43, Antonino A. Daplas wrote:
> Andi Kleen wrote:
> > On Wednesday 11 January 2006 14:05, Antonino A. Daplas wrote:
> >> Andi Kleen wrote:
> >>> On Wednesday 11 January 2006 13:24, Antonino A. Daplas wrote:
> >>>
> >>>> In the VGA console, all buffers, including scrollback is in video RAM, but
> >>>> the size is fixed and is very small.
> >>> I wonder if that can be fixed.
> >> It can be done, but it will affect VGA console performance.
> > 
> > By how much? As long as it still scrolls reasonably fast it would be ok for me.
> 
> Each character will need to be written twice, one to VGA RAM and another to
> the shadow/scrollback buffer in system RAM.

That should be basically unnoticeable. 

> It would still be reasonably fast. 
> 
> Perhaps I can implement this for vgacon.

Please do. And increase the default scrollback please or make it a CONFIG.

Thanks,
-Andi

Re: Console debugging wishlist was: Re: oops pauser.

[Jan Engelhardt]

>>> With the framebuffer console, you can increase the size of the scrollback
>>> buffer with the boot option:
>>>
>>> fbcon=scrollback:<n> (default is 32K)
>> 
>> On x86-64 vesafb is unusable slow because it does CPU scrolling cause
>> it can't use the vesa BIOS - and the others don't work everywhere. So I don't
>> think fbcon is an usable replacement.
>
>How about vga16fb + fbcon? If scrolling is slow in vga16fb, fbset -vyres 800 should
>increase performance significantly.
>

Benchmarks first.


Jan Engelhardt
--