kexec on ppc64

kexec on ppc64

[R Sharada]

Hello All,
        Sometime back, I started looking into ppc and working towards getting a
port of kexec to ppc64. As a recap (this is because I am cross-posting this mailto ppc64 and other lists as well), the idea behind kexec was to bypass the
kernel - firmware (bios/OF) interaction during bootup.
More kexec information can be obtained from Randy Dunlap's site on osdl:
http://developer.osdl.org/rddunlap/kexec
                                                                                
Some Background:
                                                                                
        Towards this, I looked at the ppc kernel boot code (primarily the prom
code that interacts with the openfirmware) and also asked a lot of questions to
Benh and Paulus regarding some of the ppc boot stuff. Ben and Paulus provided
lots of very useful information and suggestions for a clean implementation.
        One major requirement that came up for a clean implementation was to
re-order the prom_init code in ppc64. The prom_init (prom.c) code interacts withthe openfirmware to perform a lot of kernel initializations.
Since a lot of initializations were being done using firmware interaction, it
meant that for a kexec case, we would need to be able to save all of that from
the previous boot cleanly.
        Talking to Ben and Paulus, I found out that all that initialization
was not necessary at that point in boot.
        Hence, we got the idea to work on the re-order of the prom_init code.
The motivation to do this for kexec would be that it would facilitate branching
into a point after prom_init in a more cleaner setup and do away with having to
maintain a lot of kernel initialization information from the earlier boot.
                                                                                
To Dos:
                                                                                
To start with, I put together the following preliminary list for the cleanup:
        - only basic minimal initialization to be done in prom_init code. This
would include, instantiating the rtas, initializing client services, tce_table
initialization, cpu initializations, etc.
        - some other initializations, like initalization of the naca structure,
systemcfg data structures, to be moved to a early_init call, after prom_init,
but before memory management was setup, since some of the fields in systemcfg
and naca are required prior to htab/stab initializations.
        - still other initializations to be moved to a later call, maybe called
from setup_system() after kernel was relocated and memory management setup. Thiscould include setting up kernel globals like cpumask data structures (they are
currently setup in prom_init)
        - also identify relevant data (that might be required for the kexec
kernel) that could be pushed into the device_tree so that it can be passed to
the new kexec kernel
                                                                                
My current focus:
                                                                                
I am currently working on moving some of the cpumask data structures out into
setup_system and debugging some problems related to getting it working.
                                                                                
Other issues:
                                                                                
With respect to kexec, a few points came up in the discussions with Ben:
Memory accessibility in real mode in HV environment (only 256mb is accessible),
which has to be taken into account when we want to copy the new kernel. Ben
suggested that we could setup a mmu mapping making use of the HV calls, or we
have a real mode routine to copy pages.
Another point that came up was the unregistration and re-registration of the
vpa_areas in HV environment after a kexec reboot.
All this still needs to be thought through a lot more.
There are most likely to be more issues, points that have to be discussed,
sorted out and resolved to get kexec on ppc64 working.
                                                                                
Request for comments
                                                                                
I would like to solicit inputs, feedback, opinions, changes on this preliminary
idea and planned list of 'things to do'. I would also like to know the interest
to participate in this effort or anything else related to 'kexec on ppc64'.
                                                                                
Thanks and Regards,
Sharada

Re: kexec on ppc64

[Dave Hansen]

taking silly IBM list off the cc...

On Wed, 2004-07-14 at 07:45, R Sharada 
>         - also identify relevant data (that might be required for the
> kexec kernel) that could be pushed into the device_tree so that it can
> be passed to the new kexec kernel

We actually had a similar problem on x86.  The e820 table is presented
by the BIOS, and must be saved or reconstructed when you boot into the
new kernel.  

It would be really cool to have a way of passing the memory layout
information to the new kernel that is relatively cross-platform.  That
way, we don't get stuck rewriting it for each new arch.  

For instance, instead of passing the BIOS/firmware structures like LMBs 
in the device tree or e820 tables to the kexec kernel, you'd pass the
Linux concepts like zone_start_pfn and so forth.

> I would like to solicit inputs, feedback, opinions, changes on this
> preliminary idea and planned list of 'things to do'. I would also like
> to know the interest to participate in this effort or anything else
> related to 'kexec on ppc64'.

While you're ripping apart prom.c, have you thought about getting rid of
all of the RELOC() stuff?  I think Ben H. had an evil plan for that,
too.

BTW, have you seen any opportunities to replace the arch-specific things
like lmb_alloc() with normal bootmem calls?

-- Dave

Re: kexec on ppc64

[Benjamin Herrenschmidt]

On Thu, 2004-07-15 at 00:14, Dave Hansen wrote:
> taking silly IBM list off the cc...
> 
> On Wed, 2004-07-14 at 07:45, R Sharada
> >         - also identify relevant data (that might be required for the
> > kexec kernel) that could be pushed into the device_tree so that it can
> > be passed to the new kexec kernel
> 
> We actually had a similar problem on x86.  The e820 table is presented
> by the BIOS, and must be saved or reconstructed when you boot into the
> new kernel.
> 
> It would be really cool to have a way of passing the memory layout
> information to the new kernel that is relatively cross-platform.  That
> way, we don't get stuck rewriting it for each new arch.

I wouldn't even try to go that way at first. What we need is to pass
an OF device-tree around on ppc, with all the necessary informations
stuffed in it as additional properties.

> For instance, instead of passing the BIOS/firmware structures like LMBs
> in the device tree or e820 tables to the kexec kernel, you'd pass the
> Linux concepts like zone_start_pfn and so forth.

I'd prefer letting the new kernel decide those based on the low level
firmware info, or you introduce a higher level dependency between the
"old" and "new" kernels (what if those concepts change between the 2
kernels ?)

> > I would like to solicit inputs, feedback, opinions, changes on this
> > preliminary idea and planned list of 'things to do'. I would also like
> > to know the interest to participate in this effort or anything else
> > related to 'kexec on ppc64'.
> 
> While you're ripping apart prom.c, have you thought about getting rid of
> all of the RELOC() stuff?  I think Ben H. had an evil plan for that,
> too.

Paulus already removed a bunch, once we manage to completely isolate
prom_init from the rest of the kernel, it will be easier to "finish" it
by having prom_init be a totally separate link entity (could even be 32
bits relocatable code).

> BTW, have you seen any opportunities to replace the arch-specific things
> like lmb_alloc() with normal bootmem calls?

As far as lmb_alloc() is concerned, there are some constraints like beeing
in RMO for some things, or beeing at the top of the memory for others, that
aren't quite addressed by bootmem. Again, we are in the firmware env at
this point, I wouldn't rely on too much linux-specific data structures.

What we need, as we already discussed for a while with Rusty, is a specific
allocator for use within prom_init() which properly deals with both OF-side
allocations (claim the memory) to avoid walking over things like initrd,
deals with the various constraints we have, and produce a map of allocated
area as a property for use by the rest of the kernel.

As far as the post-prom-init stage is concerned, we can just turn that map
into lmb's, or try to get rid of lmb's and turn that into bootmem provided
we can deal with the various RMO constraints.

Ben.

kexec on ppc64

[Matthew McClintock]

All,

I'm trying to determine how kexec'ing works on 64 bit powerpc. When allocating a region for the kexec'ed kernel is it ever the same as the currently running kernel or do you always boot the kexec'ed kernel from a different memory region? I understand that a crash kernel will be in a different region, however I was hoping to confirm the behavior for a normal "kexec -e".

Thanks,
Matthew

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

kexec on ppc64

[Matthew McClintock]

All,

I'm trying to determine how kexec'ing works on 64 bit powerpc. When =
allocating a region for the kexec'ed kernel is it ever the same as the =
currently running kernel or do you always boot the kexec'ed kernel from =
a different memory region? I understand that a crash kernel will be in a =
different region, however I was hoping to confirm the behavior for a =
normal "kexec -e".

Thanks,
Matthew=

Re: kexec on ppc64

[Michael Neuling]

> I'm trying to determine how kexec'ing works on 64 bit powerpc. When
> allocating a region for the kexec'ed kernel is it ever the same as the
> currently running kernel or do you always boot the kexec'ed kernel
> from a different memory region? I understand that a crash kernel will
> be in a different region, however I was hoping to confirm the behavior
> for a normal "kexec -e".

The kernel will be loaded at a non zero address, but it will copy itself
to zero before it starts running.

Mikey

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec on ppc64

[Michael Neuling]

> I'm trying to determine how kexec'ing works on 64 bit powerpc. When
> allocating a region for the kexec'ed kernel is it ever the same as the
> currently running kernel or do you always boot the kexec'ed kernel
> from a different memory region? I understand that a crash kernel will
> be in a different region, however I was hoping to confirm the behavior
> for a normal "kexec -e".

The kernel will be loaded at a non zero address, but it will copy itself
to zero before it starts running.

Mikey