Faster reboots (and a better way of taking crashdumps?)

Faster reboots (and a better way of taking crashdumps?)

[Martin J. Bligh]

My real motivation for this isn't actually faster reboots,
it's rebooting at all - I have some strange hardware that
won't do init 6 in traditional ways ... but it might mean
a faster reboot for others.

What's to stop me rebooting by having machine_restart load
the first sector of the first disk (as the BIOS does), where
the LILO code should be, and just jumping to it?

1. Are there tables that are created by the BIOS that we 
destroy during Linux runtime? mps tables spring to mind - 
I can't see where we preserve them ...

2. Things that are reset by reboot that we don't reset during
normal kernel boot?

As a side effect, this means we could potentially take 
crashdumps on the way up, rather than the way down, so
the kernel is more likely to be in a working state (we'd
have to load a minimal kernel / crashdumper to take the
dump first ... this is similar to what we did with PTX).

M.

Re: Faster reboots (and a better way of taking crashdumps?)

[Jeremy Jackson]

for 2, the BIOS sets the hardware to a known state,
or if you can trigger *the* hardware reset line,
which will also do that, then you're going through
the BIOS again.  Now if you made your own bios...
see www.linuxbios.org.

there are patches where a kernel can load another
kernel, also.

As for taking crashdumps on the way up, I believe
(SGI's ?) linux kernel crash dumps does *exactly*
this.

Jeremy

----- Original Message -----
From: "Martin J. Bligh" <fletch@aracnet.com>
To: <linux-kernel@vger.kernel.org>
Sent: Friday, April 05, 2002 11:13 AM
Subject: Faster reboots (and a better way of taking crashdumps?)


> My real motivation for this isn't actually faster reboots,
> it's rebooting at all - I have some strange hardware that
> won't do init 6 in traditional ways ... but it might mean
> a faster reboot for others.
>
> What's to stop me rebooting by having machine_restart load
> the first sector of the first disk (as the BIOS does), where
> the LILO code should be, and just jumping to it?
>
> 1. Are there tables that are created by the BIOS that we
> destroy during Linux runtime? mps tables spring to mind -
> I can't see where we preserve them ...
>
> 2. Things that are reset by reboot that we don't reset during
> normal kernel boot?
>
> As a side effect, this means we could potentially take
> crashdumps on the way up, rather than the way down, so
> the kernel is more likely to be in a working state (we'd
> have to load a minimal kernel / crashdumper to take the
> dump first ... this is similar to what we did with PTX).
>
> M.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Faster reboots (and a better way of taking crashdumps?)

[Martin J. Bligh]

> for 2, the BIOS sets the hardware to a known state,
> or if you can trigger *the* hardware reset line,
> which will also do that, then you're going through
> the BIOS again.  Now if you made your own bios...
> see www.linuxbios.org.

I need to avoid going through the BIOS ... this is a 
multiquad NUMA machine, and it doesn't take kindly
to the reboot through the BIOS for various reasons.
It also takes about 4 minutes, which is a pain ;-)

I have source code access to our BIOS if I really wanted, 
I just want to avoid modifying it if possible.

> there are patches where a kernel can load another
> kernel, also.

Hmmm ... sounds interesting ... any pointers?
 
> As for taking crashdumps on the way up, I believe
> (SGI's ?) linux kernel crash dumps does *exactly*
> this.

I was under the impression that most BIOSes reset
memory on reboot, so this was impossible during a
BIOS reboot?

M.

Re: Faster reboots (and a better way of taking crashdumps?)

[Jeremy Jackson]

----- Original Message -----
From: "Martin J. Bligh" <Martin.Bligh@us.ibm.com>
Sent: Friday, April 05, 2002 5:48 PM

> I need to avoid going through the BIOS ... this is a
> multiquad NUMA machine, and it doesn't take kindly
> to the reboot through the BIOS for various reasons.
> It also takes about 4 minutes, which is a pain ;-)
>
> I have source code access to our BIOS if I really wanted,
> I just want to avoid modifying it if possible.

well keep in mind that the fastest LinuxBIOS boot is 3 seconds...
a large part of the boot time on most PCs is the BIOS setting up
DOS support and painting silly logos on the screen, all of which
can go away.  I'm guessing your NUMA system has a bit more
to do at this stage due to the hardware, but still...

>
> > there are patches where a kernel can load another
> > kernel, also.
>
> Hmmm ... sounds interesting ... any pointers?

LOBOS,

http://www.acl.lanl.gov/linuxbios/papers/index.html
http://www.usenix.org/publications/library/proceedings/als2000/minnichLOBOS.
html

two kernel monte,

http://www.scyld.com/products/beowulf/software/monte.html

There's also suspend to disk support, which is closely related.
Kind of a restartable crash dump, without the crash:

http://falcon.sch.bme.hu/~seasons/linux/swsusp.html

>
> > As for taking crashdumps on the way up, I believe
> > (SGI's ?) linux kernel crash dumps does *exactly*
> > this.
>
> I was under the impression that most BIOSes reset
> memory on reboot, so this was impossible during a
> BIOS reboot?

oss.sgi.com seems to be down today... but iirc,
it doesn't boot through bios, but stashes some critical state
in a buffer previously reserverd, uses one of the above methods
to boot a new kernel, lets this kernel do the dump, then boots
through the bios to make sure hardware is completely restored
after the crash.  I'm sure it could be tailored to suit though.

I'm currently researching combining the two, to create a LinuxBIOS
firmware debug console, which will allow complete crash dump to
be taken after a hardware reset, with the smallest possible Heisenburg
effect, aside from a hardware debugger.

Basically when the kernel panics, it dumps the CPU registers and
resets the CPU.  The firmware console makes no alterations to
the state of the hardware, instead running a modified kgdb stub
like routine, possibly without even touching RAM.  Also, if an SMI
button is available, this can be used as a hardware break switch,
allowing panics to use an even less invasive HLT instruction.

Jeremy

Re: Faster reboots - calling _start?

[Byron Stanoszek]

On Fri, 5 Apr 2002, Jeremy Jackson wrote:

> ----- Original Message -----
> From: "Martin J. Bligh" <Martin.Bligh@us.ibm.com>
> Sent: Friday, April 05, 2002 5:48 PM
> 
> > I need to avoid going through the BIOS ... this is a
> > multiquad NUMA machine, and it doesn't take kindly
> > to the reboot through the BIOS for various reasons.
> > It also takes about 4 minutes, which is a pain ;-)
> >
> > I have source code access to our BIOS if I really wanted,
> > I just want to avoid modifying it if possible.
> 
> well keep in mind that the fastest LinuxBIOS boot is 3 seconds...
> a large part of the boot time on most PCs is the BIOS setting up
> DOS support and painting silly logos on the screen, all of which
> can go away.  I'm guessing your NUMA system has a bit more
> to do at this stage due to the hardware, but still...

Wouldn't it be easier to just ljmp to the start address of the kernel in memory
(the address after the bootloader has done its thing), effectively restarting
the kernel from line 1? Or is tehre an issue with some hardware being in an
invalid state when doing this?

Maybe Eric Biederman can comment on this since he's adding new functionality to
the boot loader..

 -Byron

-- 
Byron Stanoszek                         Ph: (330) 644-3059
Systems Programmer                      Fax: (330) 644-8110
Commercial Timesharing Inc.             Email: byron@comtime.com

Re: Faster reboots - calling _start?

[Eric W. Biederman]

Byron Stanoszek <gandalf@winds.org> writes:

> On Fri, 5 Apr 2002, Jeremy Jackson wrote:
> 
> > ----- Original Message -----
> > From: "Martin J. Bligh" <Martin.Bligh@us.ibm.com>
> > Sent: Friday, April 05, 2002 5:48 PM
> > 
> > > I need to avoid going through the BIOS ... this is a
> > > multiquad NUMA machine, and it doesn't take kindly
> > > to the reboot through the BIOS for various reasons.
> > > It also takes about 4 minutes, which is a pain ;-)
> > >
> > > I have source code access to our BIOS if I really wanted,
> > > I just want to avoid modifying it if possible.
> > 
> > well keep in mind that the fastest LinuxBIOS boot is 3 seconds...
[ to login prompt ]

> > a large part of the boot time on most PCs is the BIOS setting up
> > DOS support and painting silly logos on the screen, all of which
> > can go away.  I'm guessing your NUMA system has a bit more
> > to do at this stage due to the hardware, but still...

Especially given that I can load the kernel on a dual P4 xeon system in
3 seconds from power on.  But traditional BIOS's are slow, and getting
slower for unknown reasons.

> Wouldn't it be easier to just ljmp to the start address of the kernel in memory
> (the address after the bootloader has done its thing), effectively restarting
> the kernel from line 1? Or is tehre an issue with some hardware being in an
> invalid state when doing this?
> 
> Maybe Eric Biederman can comment on this since he's adding new functionality to
> the boot loader..

I've also written the patch that allows this.

In the general case where you want to boot another version of the kernel
you must rerun the BIOS query code.  In case your new kernel can handle your
hardware better than the previous version.

The are bad cases where the kernel can leave the hardware in a state
that either confuses the BIOS or confuses the drivers when they load.
This is rare and being worked on.  But it is an independent problem.

http://download.lnxi.com/pub/src/linux-kernel-patches/kexec/linux-2.5.7.kexec.diff
http://download.lnxi.com/pub/src/mkelfImage/elfboottools-2.0.tar.gz

And despite the name I can kexec plain bzImages, now.

Eric

Re: Faster reboots - calling _start?

[Martin J. Bligh]

> Wouldn't it be easier to just ljmp to the start address of the kernel in
> memory (the address after the bootloader has done its thing), effectively
> restarting the kernel from line 1? Or is tehre an issue with some
> hardware being in an invalid state when doing this?

Two issues with that:

1. I want to be able to boot a different kernel on reboot - this
is a development machine.

2. I believe we free all the __init stuff around the end of
start_kernel, so the initial functions and data just aren't 
there any more ... of course that could be changed, but it's
both a more major change than I really want to do, and it still
doesn't solve (1) ;-)

M.

Re: Faster reboots - calling _start?

[Eric W. Biederman]

"Martin J. Bligh" <Martin.Bligh@us.ibm.com> writes:

> > Wouldn't it be easier to just ljmp to the start address of the kernel in
> > memory (the address after the bootloader has done its thing), effectively
> > restarting the kernel from line 1? Or is tehre an issue with some
> > hardware being in an invalid state when doing this?
> 
> Two issues with that:
> 
> 1. I want to be able to boot a different kernel on reboot - this
> is a development machine.
> 
> 2. I believe we free all the __init stuff around the end of
> start_kernel, so the initial functions and data just aren't 
> there any more ... of course that could be changed, but it's
> both a more major change than I really want to do, and it still
> doesn't solve (1) ;-)

Seriously check out my code it should just work unless there are special
apic shutdown rules for NUMAQ machines.

Eric

Re: Faster reboots (and a better way of taking crashdumps?)

[Suparna Bhattacharya]

Jeremy Jackson wrote:
> 
> ----- Original Message -----
> From: "Martin J. Bligh" <Martin.Bligh@us.ibm.com>
> Sent: Friday, April 05, 2002 5:48 PM
> 
> > I need to avoid going through the BIOS ... this is a
> > multiquad NUMA machine, and it doesn't take kindly
> > to the reboot through the BIOS for various reasons.
> > It also takes about 4 minutes, which is a pain ;-)
> >
> > I have source code access to our BIOS if I really wanted,
> > I just want to avoid modifying it if possible.
> 
> well keep in mind that the fastest LinuxBIOS boot is 3 seconds...
> a large part of the boot time on most PCs is the BIOS setting up
> DOS support and painting silly logos on the screen, all of which
> can go away.  I'm guessing your NUMA system has a bit more
> to do at this stage due to the hardware, but still...
> 
> >
> > > there are patches where a kernel can load another
> > > kernel, also.
> >
> > Hmmm ... sounds interesting ... any pointers?
> 
> LOBOS,
> 
> http://www.acl.lanl.gov/linuxbios/papers/index.html
> http://www.usenix.org/publications/library/proceedings/als2000/minnichLOBOS.
> html
> 
> two kernel monte,
> 
> http://www.scyld.com/products/beowulf/software/monte.html
> 
> There's also suspend to disk support, which is closely related.
> Kind of a restartable crash dump, without the crash:
> 
> http://falcon.sch.bme.hu/~seasons/linux/swsusp.html
> 
> >
> > > As for taking crashdumps on the way up, I believe
> > > (SGI's ?) linux kernel crash dumps does *exactly*
> > > this.
> >
> > I was under the impression that most BIOSes reset
> > memory on reboot, so this was impossible during a
> > BIOS reboot?
> 
> oss.sgi.com seems to be down today... but iirc,
> it doesn't boot through bios, but stashes some critical state
> in a buffer previously reserverd, uses one of the above methods
> to boot a new kernel, lets this kernel do the dump, then boots
> through the bios to make sure hardware is completely restored
> after the crash.  I'm sure it could be tailored to suit though.

What you just described is the way Mission Critical's 
crash dump facility implements this. It makes use of bootimg
Werner Almesberger's implementation of linux-boots-linux, to
soft reboot a new kernel without going through the bios on x86 
machines.

SGI lkcd doesn't do it that way today (btw, the lkcd project 
has now shifted to sourceforge, so the right place to look at 
is lkcd.sourceforge.net), but we are actually working on integrating
this mechanism from Mission Critical's mcore into lkcd, with
some enhancements/improvements. 

I'm cc'ing the lkcd-devel mailing list which is where we discuss
such stuff.

> 
> I'm currently researching combining the two, to create a LinuxBIOS
> firmware debug console, which will allow complete crash dump to
> be taken after a hardware reset, with the smallest possible Heisenburg
> effect, aside from a hardware debugger.

So how is the actual writeout accomplished ?(via LinuxBIOS ?)
 
> 
> Basically when the kernel panics, it dumps the CPU registers and
> resets the CPU.  The firmware console makes no alterations to
> the state of the hardware, instead running a modified kgdb stub
> like routine, possibly without even touching RAM.  Also, if an SMI
> button is available, this can be used as a hardware break switch,
> allowing panics to use an even less invasive HLT instruction.
> 
> Jeremy
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Faster reboots (and a better way of taking crashdumps?)

[Jeremy Jackson]

Should this go off lkml?

----- Original Message ----- 
From: "Suparna Bhattacharya" <suparna@in.ibm.com>
Sent: Monday, April 08, 2002 12:20 AM
 
> > I'm currently researching combining the two, to create a LinuxBIOS
> > firmware debug console, which will allow complete crash dump to
> > be taken after a hardware reset, with the smallest possible Heisenburg
> > effect, aside from a hardware debugger.
> 
> So how is the actual writeout accomplished ?(via LinuxBIOS ?)

well it's just an idea just now. In order to do this from code in rom,
I immagine it would just dump physical memory to a raw partition,
using polling ide drivers in LinuxBIOS.  This is probably a step
backwards, compared to modern crash dumps, but it would
allow zero alteration of memory.

It may be possible to do with a standard flash size of 128KiB,
though, which would allow virtually all motherboards to support it.

Jeremy

Re: Faster reboots (and a better way of taking crashdumps?)

[Suparna Bhattacharya]

On Wed, Apr 10, 2002 at 08:47:17PM -0700, Jeremy Jackson wrote:
> Should this go off lkml?

Should be ok to continue - I just wanted to make sure lkcd-devel 
is in the loop too, since many of us don't monitor lkml that 
closely all the time.

> 
> ----- Original Message ----- 
> From: "Suparna Bhattacharya" <suparna@in.ibm.com>
> Sent: Monday, April 08, 2002 12:20 AM
>  
> > > I'm currently researching combining the two, to create a LinuxBIOS
> > > firmware debug console, which will allow complete crash dump to
> > > be taken after a hardware reset, with the smallest possible Heisenburg
> > > effect, aside from a hardware debugger.
> > 
> > So how is the actual writeout accomplished ?(via LinuxBIOS ?)
> 
> well it's just an idea just now. In order to do this from code in rom,
> I immagine it would just dump physical memory to a raw partition,
> using polling ide drivers in LinuxBIOS.  This is probably a step

OK. There have been plans to do this via polling drivers in software,
but guess if you can handle it at the LinuxBIOS level the impact may 
be still lower. 

> backwards, compared to modern crash dumps, but it would
> allow zero alteration of memory.
> 
> It may be possible to do with a standard flash size of 128KiB,
> though, which would allow virtually all motherboards to support it.
> 
> Jeremy

Re: Faster reboots (and a better way of taking crashdumps?)

[Alan Cox]

> What's to stop me rebooting by having machine_restart load
> the first sector of the first disk (as the BIOS does), where
> the LILO code should be, and just jumping to it?

In theory nothing

> 1. Are there tables that are created by the BIOS that we 
> destroy during Linux runtime? mps tables spring to mind - 
> I can't see where we preserve them ...

They should be in E820 reserved pages anyway and we do keep them and the
EBDA safe. You will however have blown away ACPI pages marked as disposable

> 2. Things that are reset by reboot that we don't reset during
> normal kernel boot?

Possibly. I wouldnt like to hand control back to the BIOS but the kernel
ought to be ok with itself.

Alan

Re: Faster reboots (and a better way of taking crashdumps?)

[Martin J. Bligh]

 1. Are there tables that are created by the BIOS that we 
>> destroy during Linux runtime? mps tables spring to mind - 
>> I can't see where we preserve them ...
> 
> They should be in E820 reserved pages anyway and we do keep them 
> and the EBDA safe. 

Ah, OK. I will have to check the BIOS is doing this correctly,
since I hacked it to move the MPS tables to a different place
(below 8Mb). I should really fix that using a fixmap or something
anyway ...

> You will however have blown away ACPI pages marked as disposable

Pah, ACPI ;-) I don't have ACPI on these machines, but it would
be needed for a more general solution - sounds easy enough to fix 
anyway - we just keep them and mark them reserved during the Linux
ACPI parse, I think.

Thanks,

M.

Re: Faster reboots (and a better way of taking crashdumps?)

[Eric W. Biederman]

"Martin J. Bligh" <fletch@aracnet.com> writes:

> My real motivation for this isn't actually faster reboots,
> it's rebooting at all - I have some strange hardware that
> won't do init 6 in traditional ways ... but it might mean
> a faster reboot for others.
> 
> What's to stop me rebooting by having machine_restart load
> the first sector of the first disk (as the BIOS does), where
> the LILO code should be, and just jumping to it?

Be very careful with loading a boot sector.  The problem is
that lilo will ask the BIOS to drive the disk, and the disk
is almost certainly in a different state than when the BIOS left it,
and the BIOS hasn't been given a reset state command.  Without letting
the BIOS know you did something strange you are going out and looking
for trouble.

But if you can load a boot sector you can just about as easily load
the whole kernel, which on startup will only ask the BIOS hardware
information and not to drive the hardware (which should be safe).


> 1. Are there tables that are created by the BIOS that we 
> destroy during Linux runtime? mps tables spring to mind - 
> I can't see where we preserve them ...

Generally MPS tables are in regions of memory that we preserve anyway.

> 2. Things that are reset by reboot that we don't reset during
> normal kernel boot?

A sane BIOS will toggle the board level reset line on reboot.
The all don't but that makes it look like a fresh boot, with
a negligible speed penalty.

> As a side effect, this means we could potentially take 
> crashdumps on the way up, rather than the way down, so
> the kernel is more likely to be in a working state (we'd
> have to load a minimal kernel / crashdumper to take the
> dump first ... this is similar to what we did with PTX).

I guess if you were careful you could.  The fact that you can't rely
on the BIOS to drive the hardware is significant though.

Eric