Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

Hi,

On Wed, Jan 26, 2011 at 06:25:19AM +1000, Linus Torvalds wrote:
> On Tue, Jan 25, 2011 at 11:47 PM, Ahmed S. Darwish <darwish.07@gmail.com> wrote:
> >
> > As a possible solution, below patches prototypes the idea of persistently
> > storing the kernel log ring to a hard disk partition using the enhanced BIOS
> > 0x13 services.
> >
>
> Quite frankly, I'm not likely to _ever_ merge anything like this.
> 
> Over the years, many people have tried to write things to disk on
> oops. I refuse to take it. No way in hell do I want the situation of
> "the system is screwed, so let's overwrite the disk" to be something
> the kernel I release might do. It's crazy. That disk is a lot more
> important than the kernel, and overwriting it when we might have
> serious memory corruption issues or something is not a thing I feel is
> appropriate.
> 

OK, I've carefully read all the criticism posted here and in the other
replies. A short summary:

- It will be _too_ much to restore the PIC, IO-APIC, PCI configuration,
  and disk controllers to the BIOS Power-on self-test (POST) state. The
  solution will not be guaranteed to work on all machines.

- The write path of the BIOS INT 0x13 services is not tested enough by
  contemporary x86 kernels (Windows and Linux). Thus, that part of the
  BIOS is probably not as stable as I originally claimed.

Possible solutions:

- Having a minimal "fail-safe" disk controller driver to access the
  disk. This can't be made "perfect" enough to work on all combinations
  of machines (Though it can work on a small subset, e.g. the usual lot
  of x86 laptops with Intel chipsets). Linus is not OK with the _whole_
  idea of doing _any_ disk write on panic.

- Use kexec/kdump. I wonder though, can kexec work on early panics? The
  kind where the disk itself might not yet be initialized?

  If not, can kexec be modified to do so? That possibly needs the help
  of a a bootloader extension.

- The latest approach (proposed by Linus) is to forget the disk: jump to
  real-mode, but display the kernel log in a fancy format (with scroll
  ups and downs) instead.

  Will re-initializing the VGA registers to their POST state be possible?
  If not, what about a "fail-safe" VGA driver?

I'm most likely going to implement either the second or the third point,
so I'd really appreciate some input.

thanks,

> 
> If you want to do the BIOS services thing, do it for video: copy the
> oops to low RAM, return to real mode, re-run the graphics card POST
> routines to initialize text-mode, and use the BIOS to print out the
> oops.  That is WAY less scary than writing to disk.
> 
>                               Linus

-- 
Darwish
http://darwish.07.googlepages.com

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ingo Molnar]

* Ahmed S. Darwish <darwish.07@gmail.com> wrote:

> - The latest approach (proposed by Linus) is to forget the disk: jump to
>   real-mode, but display the kernel log in a fancy format (with scroll
>   ups and downs) instead.
> 
>   Will re-initializing the VGA registers to their POST state be possible?
>   If not, what about a "fail-safe" VGA driver?
> 
> I'm most likely going to implement either the second or the third point,
> so I'd really appreciate some input.

The third one suggested by Linus is the most useful and most direct one IMO. Such a 
'oops mode' screen would be useful for regular kernel crashes as well.

Also, have you tried BIOS warm reset vector, which is supposed to reboot without 
clearing RAM contents - how well does it work in practice on typical laptops? If on 
crash we could reboot without memory getting cleared that would open up a vast area 
of space to store the kernel log into (RAM).

Thanks,

	Ingo

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

On Wed, Jan 26, 2011 at 01:58:13PM +0100, Ingo Molnar wrote:
> 
> * Ahmed S. Darwish <darwish.07@gmail.com> wrote:
> 
> > - The latest approach (proposed by Linus) is to forget the disk: jump to
> >   real-mode, but display the kernel log in a fancy format (with scroll
> >   ups and downs) instead.
> > 
> >   Will re-initializing the VGA registers to their POST state be possible?
> >   If not, what about a "fail-safe" VGA driver?
> > 
> > I'm most likely going to implement either the second or the third point,
> > so I'd really appreciate some input.
> 
> The third one suggested by Linus is the most useful and most direct one IMO. Such a 
> 'oops mode' screen would be useful for regular kernel crashes as well.
> 

Yes, it is the safest approach.

If I can make kexec and kdump work on early panics using some help from
bootloaders though, that can possibly give us the feature of saving kernel
logs to disk upon panics _without_ danger. I did not look at the kexec and
kdump code bases yet, so this might be a pipe dream (respective developers
are CCed -- input appreciated).

>
> Also, have you tried BIOS warm reset vector, which is supposed to reboot without 
> clearing RAM contents - how well does it work in practice on typical laptops? If on 
> crash we could reboot without memory getting cleared that would open up a vast area 
> of space to store the kernel log into (RAM).
> 

AFAIK, the lmode->rmode transition is more forward-compatible.

It seems the only place warm boot was documented is in the Intel MP spec,
a 12-years old document long obsoleted by ACPI. Meanwhile, the real-mode
transition is rigorously documented in the current Intel and AMD manuals,
albeit in kind of a holier-than-thou approach.

I'll investigate this further though.

thanks,

-- 
Darwish
http://darwish.07.googlepages.com

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ingo Molnar]

* Ahmed S. Darwish <darwish.07@gmail.com> wrote:

> > Also, have you tried BIOS warm reset vector, which is supposed to reboot without 
> > clearing RAM contents - how well does it work in practice on typical laptops? If 
> > on crash we could reboot without memory getting cleared that would open up a 
> > vast area of space to store the kernel log into (RAM).
> 
> AFAIK, the lmode->rmode transition is more forward-compatible.
> 
> It seems the only place warm boot was documented is in the Intel MP spec, a 
> 12-years old document long obsoleted by ACPI. Meanwhile, the real-mode transition 
> is rigorously documented in the current Intel and AMD manuals, albeit in kind of a 
> holier-than-thou approach.

I mean, use the warm reset vector to truly reset the box.

Then, once a stable known-good kernel boots, *that* kernel can then recover all the 
log data which is sitting in a well-known place in RAM, automatically and 
transparently.

Basically a bit like kexec, just more convenient and it also goes through the BIOS 
warm reset, so it might work better than kexec ...

Thanks,

	Ingo

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[H. Peter Anvin]

On 01/26/2011 06:00 AM, Ingo Molnar wrote:
>
> I mean, use the warm reset vector to truly reset the box.
>
> Then, once a stable known-good kernel boots, *that* kernel can then recover all the
> log data which is sitting in a well-known place in RAM, automatically and
> transparently.
>
> Basically a bit like kexec, just more convenient and it also goes through the BIOS
> warm reset, so it might work better than kexec ...
>

The problem is that the BIOS will often wipe memory.

	-hpa

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> On 01/26/2011 06:00 AM, Ingo Molnar wrote:
> >
> >I mean, use the warm reset vector to truly reset the box.
> >
> >Then, once a stable known-good kernel boots, *that* kernel can then recover all the
> >log data which is sitting in a well-known place in RAM, automatically and
> >transparently.
> >
> >Basically a bit like kexec, just more convenient and it also goes through the BIOS
> >warm reset, so it might work better than kexec ...
> >
> 
> The problem is that the BIOS will often wipe memory.

Yes - but at least a long time ago there used to be modi of reboot when the BIOS did 
not do this. Whether that's possible at all with modern BIOSen is a big question ...

Thanks,

	Ingo

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Eric W. Biederman]

Ingo Molnar <mingo@elte.hu> writes:

> * H. Peter Anvin <hpa@zytor.com> wrote:
>
>> On 01/26/2011 06:00 AM, Ingo Molnar wrote:
>> >
>> >I mean, use the warm reset vector to truly reset the box.
>> >
>> >Then, once a stable known-good kernel boots, *that* kernel can then recover all the
>> >log data which is sitting in a well-known place in RAM, automatically and
>> >transparently.
>> >
>> >Basically a bit like kexec, just more convenient and it also goes through the BIOS
>> >warm reset, so it might work better than kexec ...
>> >
>> 
>> The problem is that the BIOS will often wipe memory.
>
> Yes - but at least a long time ago there used to be modi of reboot when the BIOS did 
> not do this. Whether that's possible at all with modern BIOSen is a
> big question ...

As I recall that was how you went from 32bit mode to 16bit real mode.
The issue is that mode never reset the hardware it simply changed the
cpu mode.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[H. Peter Anvin]

On 01/27/2011 07:47 AM, Eric W. Biederman wrote:
>>
>> Yes - but at least a long time ago there used to be modi of reboot when the BIOS did
>> not do this. Whether that's possible at all with modern BIOSen is a
>> big question ...
>
> As I recall that was how you went from 32bit mode to 16bit real mode.
> The issue is that mode never reset the hardware it simply changed the
> cpu mode.
>

Quite correct.  It doesn't just bypass the memory wipe, it bypasses the 
entire POST.

	-hpa

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Luck, Tony]

>- The latest approach (proposed by Linus) is to forget the disk: jump to
>  real-mode, but display the kernel log in a fancy format (with scroll
>  ups and downs) instead.

A while ago (first Plumbers conference?) someone was talking about
using a 2-d barcode to display the tail of the kernel log & oops
register data - with the plan that you could capture the image with
a cell phone camera, and then get all the oops data without worrying
about transcription errors as you wrote down & re-typed all the hex.

Anyone know what happened to that plan?

-Tony



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Dave Jones]

On Wed, Jan 26, 2011 at 03:07:44PM -0800, Luck, Tony wrote:
 > >- The latest approach (proposed by Linus) is to forget the disk: jump to
 > >  real-mode, but display the kernel log in a fancy format (with scroll
 > >  ups and downs) instead.
 > 
 > A while ago (first Plumbers conference?) someone was talking about
 > using a 2-d barcode to display the tail of the kernel log & oops
 > register data - with the plan that you could capture the image with
 > a cell phone camera, and then get all the oops data without worrying
 > about transcription errors as you wrote down & re-typed all the hex.
 > 
 > Anyone know what happened to that plan?

How is this more useful than a photograph of the backtrace ?

	Dave


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Luck, Tony]

> How is this more useful than a photograph of the backtrace ?

You can fit a lot more data into the 2-d barcode that will fit on
the screen.  You can also automate the recovery of the data (e.g.
for posting to kerneloops.org).

-Tony

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

On Wed, Jan 26, 2011 at 03:07:44PM -0800, Luck, Tony wrote:
> >- The latest approach (proposed by Linus) is to forget the disk: jump to
> >  real-mode, but display the kernel log in a fancy format (with scroll
> >  ups and downs) instead.
> 
> A while ago (first Plumbers conference?) someone was talking about
> using a 2-d barcode to display the tail of the kernel log & oops
> register data - with the plan that you could capture the image with
> a cell phone camera, and then get all the oops data without worrying
> about transcription errors as you wrote down & re-typed all the hex.
> 

Yes, I added a link to the video of this presentation in PATCH #0/2.

--
Darwish

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

(Very timely! was just exploring kexec-tools and friends)

On Wed, Jan 26, 2011 at 09:13:38PM -0500, Vivek Goyal wrote:
> On Wed, Jan 26, 2011 at 02:49:54PM +0200, Ahmed S. Darwish wrote:
> > 
> >   If not, can kexec be modified to do so? That possibly needs the help
> >   of a a bootloader extension.
> 
> - Generally kdump uses same kernel as first one. So if first kernel panics
>   then kdump kernel will also panic. Though one can keep a known good kdump
>   kernel.
> 

I was thinking of a 'special' kdump kernel: the older kernel will just pass
the physical RAM address of its ring buffer to the newer kernel. The newer
kernel will be __so specialized__ that it will just:

 - display the log in a friendly manner
 - write it to the swap partition
 - .. and halt

This way, I completely toss the BIOS while accomplishing similar goals.

>
> - Currently kexec-tools take care of loading the kdump kernel, and then
>   preparing memory map for second kernel and pass to it and also pass
>   along some information about vmcore in ELF headers to second kernel. 
>
> There is so much to it that I really don't think that modifying bootloaders to
> do all that might not trivial and more importantly I think it probably is not
> worth the effort.
> 

IMHO if done in a user-friendly and stable manner, it can really be worth
the effort. But quite honestly, since I did not read all the setup work
done by kexec-tools yet, I might be too optimistic.

> Thanks
> Vivek

thanks,

-- 
Darwish
http://darwish.07.googlepages.com

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[H. Peter Anvin]

On 01/26/2011 06:13 PM, Vivek Goyal wrote:
>>
>> - Use kexec/kdump. I wonder though, can kexec work on early panics? The
>>    kind where the disk itself might not yet be initialized?
>
> As of today kexec/kdump does not work with early panics as kdump kernel
> needs to be loaded in reserved memory and that action happens once the
> first kernel has booted.
>
> There is so much to it that I really don't think that modifying bootloaders to
> do all that might not trivial and more importantly I think it probably is not
> worth the effort.
>

It's really not all that hard, actually.

	-hpa

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

On Wed, Jan 26, 2011 at 08:44:14PM -0800, H. Peter Anvin wrote:
> On 01/26/2011 06:13 PM, Vivek Goyal wrote:
>>>
>>> - Use kexec/kdump. I wonder though, can kexec work on early panics? The
>>>    kind where the disk itself might not yet be initialized?
>>
>> As of today kexec/kdump does not work with early panics as kdump kernel
>> needs to be loaded in reserved memory and that action happens once the
>> first kernel has booted.
>>
>> There is so much to it that I really don't think that modifying bootloaders to
>> do all that might not trivial and more importantly I think it probably is not
>> worth the effort.
>>
>
> It's really not all that hard, actually.
>

Nice, I'll try prototyping this using syslinux and kexec: this seems the
safest path so far. Further details are in my first reply to Vivek.

> 	-hpa

thanks,

-- 
Darwish
http://darwish.07.googlepages.com

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Eric W. Biederman]

"Ahmed S. Darwish" <darwish.07@gmail.com> writes:

> On Wed, Jan 26, 2011 at 08:44:14PM -0800, H. Peter Anvin wrote:
>> On 01/26/2011 06:13 PM, Vivek Goyal wrote:
>>>>
>>>> - Use kexec/kdump. I wonder though, can kexec work on early panics? The
>>>>    kind where the disk itself might not yet be initialized?
>>>
>>> As of today kexec/kdump does not work with early panics as kdump kernel
>>> needs to be loaded in reserved memory and that action happens once the
>>> first kernel has booted.
>>>
>>> There is so much to it that I really don't think that modifying bootloaders to
>>> do all that might not trivial and more importantly I think it probably is not
>>> worth the effort.
>>>
>>
>> It's really not all that hard, actually.
>>
>
> Nice, I'll try prototyping this using syslinux and kexec: this seems the
> safest path so far. Further details are in my first reply to Vivek.

Given the inertia of boot loaders it might make sense to compile in the
crash recovery code.

Regardless the way I would organize this is simply to have a chunk of
memory that you place the crash reporting kernel and initrd in, and
simply tell the linux about it, at boot time.  Then kexec on panic can
be pointed at it and that is the end of the matter.  That is very simple.

It might be a bit tricky to get all of the context information the crash
kernel needs, but it should not be too bad.  I am with Vivek in being a
bit dubious if this will do better in practice than having a crash
kernel that we load from the initramfs, simply because the maintenance
difficulty of a specialized component.  But I see no reason to not add
the basic mechanism, to the kernel and bootloaders.  The basic mechanism
is simple and easy to maintain, and ultimately should work across platforms.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to disk upon panic

[Ahmed S. Darwish]

Hi,

Quick note: the Internet has just returned back here after a full
5-day shutdown by the “authorities”. I will hopefully return back
home on Saturday to continue working on this.

thanks,

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec