Patch boot/common/relocate.S to start kernel at 0x000c

Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

arch/ppc/boot/common/relocate.S starts the kernel at address 0
which fails if you have a second cpu running at secondary_hold.
(The code at secondary hold writes the cpu id at address 4.)

This patch starts the kernel at address 0xc instead.

Thanks,
-Dale

===== arch/ppc/boot/common/relocate.S 1.6 vs edited =====
--- 1.6/arch/ppc/boot/common/relocate.S	Thu Feb 27 12:40:14 2003
+++ edited/arch/ppc/boot/common/relocate.S	Tue Mar  4 10:09:52 2003
@@ -200,8 +200,10 @@

 	/*
 	 * Start at the begining.
+	 * Well, actually on the 4th instruction since we
+	 * overwrite the first 3 sometimes (which are 'nop').
 	 */
-	li	r9,0x0000
+	li	r9,0x000c
 	mtlr	r9
 	blr


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dan Malek]

Dale Farnsworth wrote:

> arch/ppc/boot/common/relocate.S starts the kernel at address 0
> which fails if you have a second cpu running at secondary_hold.
> (The code at secondary hold writes the cpu id at address 4.)

Ummm....are you _sure_ this patch works for all possible boards
that use this bootloader file?  We've been jumping everyone to
location zero for a long time.

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Tue, Mar 04, 2003 at 07:24:03PM +0000, Dan Malek wrote:
> Dale Farnsworth wrote:
>
> >arch/ppc/boot/common/relocate.S starts the kernel at address 0
> >which fails if you have a second cpu running at secondary_hold.
> >(The code at secondary hold writes the cpu id at address 4.)
>
> Ummm....are you _sure_ this patch works for all possible boards
> that use this bootloader file?  We've been jumping everyone to
> location zero for a long time.

No, I wasn't sure.  Now after some more looking, I'm sure that
it *won't* work for 40x, 44x, and 8xx.

I'll find another way.

Thanks,
-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dan Malek]

Dale Farnsworth wrote:

> No, I wasn't sure.  Now after some more looking, I'm sure that
> it *won't* work for 40x, 44x, and 8xx.
>
> I'll find another way.

Well, one option that may be the easiest is to just change all
of the linux head.S files to have the three nops again.  When we
split these file apart a while back, we just removed everything that
wasn't necessary.  Just put a note in those files why they are
present.  They don't cost anything since these are exception vector
locations or temporary storage locations for special purposes.

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Tue, Mar 04, 2003 at 08:21:03PM +0000, Dan Malek wrote:
> Well, one option that may be the easiest is to just change all
> of the linux head.S files to have the three nops again.  When we
> split these file apart a while back, we just removed everything that
> wasn't necessary.  Just put a note in those files why they are
> present.  They don't cost anything since these are exception vector
> locations or temporary storage locations for special purposes.

What about this approach?

The problem is that on some systems, the cpu 1 runs at
__secondary_hold before cpu 0 runs at _start and __secondary_hold
overwrites the nop at 0x4.

If we branch around the the the word at 0x4, the kernel can start
at 0 and we don't need to modify the other head.S files.

The following patch works on my SMP system.

Thanks,
-Dale

===== head.S 1.80 vs edited =====
--- 1.80/arch/ppc/kernel/head.S	Thu Feb 27 12:40:15 2003
+++ edited/head.S	Tue Mar  4 15:52:24 2003
@@ -88,8 +88,13 @@
 	 * but we're always started by some kind of bootloader now.
 	 *  -- Cort
 	 */
-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
+	/*
+	 * The first two words are used by __secondary_hold on SMP systems.
+	 * The second word may be overwritten before the first cpu runs
+	 * here, so we branch around it.
+	 */
+	b	__start
+	nop
 	nop

 /* PMAC

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dan Malek]

Dale Farnsworth wrote:

> -	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> -	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> +	/*
> +	 * The first two words are used by __secondary_hold on SMP systems.
> +	 * The second word may be overwritten before the first cpu runs
> +	 * here, so we branch around it.
> +	 */
> +	b	__start
> +	nop
>  	nop

I thought of this, but the comment on the first nop bothered me.
Was it just a cut/paste without updating the comment, or could the nop
and location zero be used by MP systems just like the one at location 0x04?

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Tue, Mar 04, 2003 at 11:21:30PM +0000, Dan Malek wrote:
> Dale Farnsworth wrote:
>
> >-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> >-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> >+	/*
> >+	 * The first two words are used by __secondary_hold on SMP systems.
> >+	 * The second word may be overwritten before the first cpu runs
> >+	 * here, so we branch around it.
> >+	 */
> >+	b	__start
> >+	nop
> > 	nop
>
> I thought of this, but the comment on the first nop bothered me.
> Was it just a cut/paste without updating the comment, or could the nop
> and location zero be used by MP systems just like the one at location 0x04?

In __secondary_hold, location 0x4 is written by secondary cpus, and
location 0x0 is read, so I guess the comment is accurate, that both
locations are "used".  Cpu 0 writes location 0 to tell the secondary
cpus they can continue.

I know the above code works for my dual-7455 system.

Thanks,
-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Tom Rini]

On Tue, Mar 04, 2003 at 06:21:30PM -0500, Dan Malek wrote:
>
> Dale Farnsworth wrote:
>
> >-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> >-	nop	/* used by __secondary_hold on prep (mtx) and chrp smp */
> >+	/*
> >+	 * The first two words are used by __secondary_hold on SMP systems.
> >+	 * The second word may be overwritten before the first cpu runs
> >+	 * here, so we branch around it.
> >+	 */
> >+	b	__start
> >+	nop
> > 	nop
>
> I thought of this, but the comment on the first nop bothered me.
> Was it just a cut/paste without updating the comment, or could the nop
> and location zero be used by MP systems just like the one at location 0x04?

Thinking waaaay back, it was probably me doing something wrong. :)  The
easy way to verify what really happens is to check out the MP code in
prep/misc.c.  I would have thought that chrp smp would do the pmac thing
of calling back to the fw, but I could be wrong here..

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Tue, Mar 04, 2003 at 11:42:53PM +0000, Tom Rini wrote:
> Thinking waaaay back, it was probably me doing something wrong. :)

That was my first thought, since prep/head.S starts the kernel at 0xc. :)

> The easy way to verify what really happens is to check out the MP code in
> prep/misc.c.  I would have thought that chrp smp would do the pmac thing
> of calling back to the fw, but I could be wrong here..

prep/misc.c just starts the secondary cpu at 0xc0, __secondary_hold.
It's ok for that to happen before cpu 0 runs the kernel, since
prep/head.S runs it at 0xc.

Pmacs also start the secondary at __secondary_hold, but after cpu 0 is
already running the kernel, so it's not an issue.

It looks to me like all ppc smp systems start their secondary cpus
at __secondary_hold.  (I'd like to know if I'm wrong about that.)
Since __secondary_hold writes to 0x4 and only reads 0x0 looking for
it's cpu id, I still like the "b __start" at 0x0.

Thanks,
-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Paul Mackerras]

Dale Farnsworth writes:

> Since __secondary_hold writes to 0x4 and only reads 0x0 looking for
> it's cpu id, I still like the "b __start" at 0x0.

Why are we still using locations 0 and 4 if that is causing trouble?
Why not change to 0xf0 and 0xf4 or something else out of the way?  We
can do the change simultaneously in the boot wrapper and in head.S, so
there are no compatibility issues.

Just a suggestion. :)

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Wolfgang Denk]

In message <15973.18737.651334.824024@argo.ozlabs.ibm.com> you wrote:
>
> Why are we still using locations 0 and 4 if that is causing trouble?
> Why not change to 0xf0 and 0xf4 or something else out of the way?  We

Because then you might step on the abatron_pteptrs area.


Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Whom the gods would destroy, they first teach BASIC.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Wolfgang Denk]

In message <15973.18737.651334.824024@argo.ozlabs.ibm.com> you wrote:
>
> Why are we still using locations 0 and 4 if that is causing trouble?
> Why not change to 0xf0 and 0xf4 or something else out of the way?  We

BTW: _all_ this writing to kernel text is a Bad  Thing  (TM),  as  it
makes XIP (for example running from ROM) really difficult.


Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Of all the things I've lost, I miss my mind the most.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Wed, Mar 05, 2003 at 02:24:32AM +0000, Paul Mackerras wrote:
> Wolfgang Denk writes:
>
> > BTW: _all_ this writing to kernel text is a Bad  Thing  (TM),  as  it
> > makes XIP (for example running from ROM) really difficult.
>
> Well, there is no reason why the secondary hold code shouldn't just
> reference an ordinary variable in the kernel data section.  It would
> take a couple more instructions, but that shouldn't be a problem.  The
> starting address for the secondary hold code needs to be well-known
> (at least to the prep boot wrapper) but the variable that it spins on
> doesn't.
>
> Paul.

Well, that does add the requirement that kernel data be mapped at its
linked-at address before __secondary_hold references said variable.

By the way, the recent addition of the call to smp_call_init_cpu,
places the same requirement on __secondary_start, unfortunately.

-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Wed, Mar 05, 2003 at 02:21:35AM +0000, Troy Benjegerdes wrote:
> If you can find a SMP system that uses XIP, then let's change it.
>
> We should only be writing to kernel text on an SMP system that has
> firmware that requires you to jump into the wrapper on both cpus.

I believe it happens on all ppc SMP systems.  It certainly happens
on powermacs which don't use the wrapper for cpu != 0.

> Dale, instead of changing head.s, can you use your own relocate.s for
> your wrapper? Don't you have to mess around with holding the 2nd cpu in
> the wrapper?

Sure, we could add an ifdef to have the wrapper start the kernel at 0xc like
prep/head.S does.  I'd rather use common code where possible though.

-Dale

P.S.  Note that I'm not proposing any additional writes to low memory.
I'm only proposing that we change the nop at 0x0 to "b .+0xc" .

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Benjamin Herrenschmidt]

> Well, that does add the requirement that kernel data be mapped at its
> linked-at address before __secondary_hold references said variable.
>
> By the way, the recent addition of the call to smp_call_init_cpu,
> places the same requirement on __secondary_start, unfortunately.

smp_call_init_cpu() is broken in various ways, and I'm about to kill
it, sorry for the inconvenient. I'm finishing tests of the replacement
in my tree now (and already pushed it to 2.5).

However, I'm replacing it with some cleaner assembly code that still
has to access the kernel memory. secondary_start has to be called
with MSR:DR=0 and kernel at it' linked location (but the later shouldn't
be a problem, and clearing MSR:DR neither).

I'll push the new code today

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Tom Rini]

On Tue, Mar 04, 2003 at 08:21:35PM -0600, Troy Benjegerdes wrote:

> Dale, instead of changing head.s, can you use your own relocate.s for
> your wrapper? Don't you have to mess around with holding the 2nd cpu in
> the wrapper?

Ack, no.  I'd rather see if #ifdef CONFIG_SMP first, but on the whole I
think there's a better solution for everyone.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Tom Rini]

On Wed, Mar 05, 2003 at 01:24:32PM +1100, Paul Mackerras wrote:
> Wolfgang Denk writes:
>
> > BTW: _all_ this writing to kernel text is a Bad  Thing  (TM),  as  it
> > makes XIP (for example running from ROM) really difficult.
>
> Well, there is no reason why the secondary hold code shouldn't just
> reference an ordinary variable in the kernel data section.  It would
> take a couple more instructions, but that shouldn't be a problem.  The
> starting address for the secondary hold code needs to be well-known
> (at least to the prep boot wrapper) but the variable that it spins on
> doesn't.

Keep in mind that this issue has come up because there will be more
non-CONFIG_ALL_PPC SMP systems being supported.

Dale, how hard would it be to change the code you have (and the PReP
code as well which maybe we can get Troy to test :)) to do what Paul
suggested above and use an ordinary variable in the kernel data section?

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dan Malek]

Wolfgang Denk wrote:

> BTW: _all_ this writing to kernel text is a Bad  Thing  (TM),  as  it
> makes XIP (for example running from ROM) really difficult.

Well, technically you aren't writing to kernel text space, you are
writing to low real memory locations. :-)  I've done several PowerPC XIP
from rom systems without any trouble, so I kind of fail to understand
why this would cause trouble.  I don't know what we gain from XIP,
except a little available ram space, but it people want it.....


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Wed, Mar 05, 2003 at 02:24:32AM +0000, Paul Mackerras wrote:
> Well, there is no reason why the secondary hold code shouldn't just
> reference an ordinary variable in the kernel data section.  It would
> take a couple more instructions, but that shouldn't be a problem.  The
> starting address for the secondary hold code needs to be well-known
> (at least to the prep boot wrapper) but the variable that it spins on
> doesn't.
>
> Paul.

I took a stab at implementing this.  It's simple when the kernel text
and data are relocated as a unit and PIC references work.  Unfortunately,
that doesn't always happen.  On OF machines, prom_hold_cpus() in
prom_init.c only copies the first 0x100 bytes of kernel text to
location 0 and starts the secondary cpus running at secondary_hold()
within that area.  I thought of passing in a pointer to kernel data,
but that's not sufficient because it looks like in some cases kernel
data is relocated while secondary_hold() is running.

So, if kernel data may be relocated while secondary_hold is polling,
the polled location can't be in kernel data.  Location 0x0 and 0x4 are
beginning to look good to me.

Does this seem right, or am I misreading the code?

-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Tue, Mar 04, 2003 at 08:21:03PM +0000, Dan Malek wrote:
> Dale Farnsworth wrote:
>
> >No, I wasn't sure.  Now after some more looking, I'm sure that
> >it *won't* work for 40x, 44x, and 8xx.
> >
> >I'll find another way.
>
> Well, one option that may be the easiest is to just change all
> of the linux head.S files to have the three nops again.  When we
> split these file apart a while back, we just removed everything that
> wasn't necessary.  Just put a note in those files why they are
> present.  They don't cost anything since these are exception vector
> locations or temporary storage locations for special purposes.

I didn't find a better way.
This patch does the following:

Avoid using the first 3 locations of kernel text, since they
may be overwritten by SMP startup code before the kernel is
started.  Modify the common bootwrapper code to start the kernel
at 0x000c.  Add nops at the beginning of teh various head*.S files.

Modified files:
arch/ppc/boot/common/relocate.S
	Start the kernel at 0x000c, since the first few instructions
	may be overwritten by SMP startup code.

arch/ppc/kernel/head_440.S
	Place three nop instructions at _stext since the bootloader
	now starts the kernel at 0x000c.

arch/ppc/kernel/head_4xx.S
	Place three nop instructions at _stext since the bootloader
	now starts the kernel at 0x000c.

arch/ppc/kernel/head_8xx.S
	Place three nop instructions at _stext since the bootloader
	now starts the kernel at 0x000c.

arch/ppc/kernel/iSeries_head.S
	Place three nop instructions at _stext since the bootloader
	now starts the kernel at 0x000c.

Please consider applying.

Thanks,
-Dale

===== arch/ppc/boot/common/relocate.S 1.6 vs edited =====
--- 1.6/arch/ppc/boot/common/relocate.S	Thu Feb 27 12:40:14 2003
+++ edited/arch/ppc/boot/common/relocate.S	Thu Mar 13 15:00:03 2003
@@ -199,9 +197,11 @@
 	li	r6,0

 	/*
-	 * Start at the begining.
+	 * Start at the beginning.
+	 * Well, actually on the 4th instruction since we
+	 * sometimes overwrite the first 3 (which are 'nop').
 	 */
-	li	r9,0x0000
+	li	r9,0x000c
 	mtlr	r9
 	blr

===== arch/ppc/kernel/head_440.S 1.13 vs edited =====
--- 1.13/arch/ppc/kernel/head_440.S	Mon Feb 24 16:26:48 2003
+++ edited/arch/ppc/kernel/head_440.S	Thu Mar 13 14:56:17 2003
@@ -58,6 +58,11 @@
 	 * of abatron_pteptrs
 	 */
 	nop
+	/* To accomodate some SMP systems that overwrite the first few
+	 * locations before cpu 0 starts, the bootloader starts us at 0xc.
+	 */
+	nop
+	nop
 /*
  * Save parameters we are passed
  */
===== arch/ppc/kernel/head_4xx.S 1.42 vs edited =====
--- 1.42/arch/ppc/kernel/head_4xx.S	Thu Feb 27 12:40:15 2003
+++ edited/arch/ppc/kernel/head_4xx.S	Thu Mar 13 14:56:19 2003
@@ -62,6 +62,13 @@
 _GLOBAL(_stext)
 _GLOBAL(_start)

+	/* To accomodate some SMP systems that overwrite the first few
+	 * locations before cpu 0 starts, the bootloader starts us at 0xc.
+	 */
+	nop
+	nop
+	nop
+
 	/* Save parameters we are passed.
 	*/
 	mr	r31,r3
===== arch/ppc/kernel/head_8xx.S 1.41 vs edited =====
--- 1.41/arch/ppc/kernel/head_8xx.S	Thu Feb 27 12:40:15 2003
+++ edited/arch/ppc/kernel/head_8xx.S	Thu Mar 13 14:56:20 2003
@@ -79,6 +79,12 @@

 	.globl	__start
 __start:
+	/* To accomodate some SMP systems that overwrite the first few
+	 * locations before cpu 0 starts, the bootloader starts us at 0xc.
+	 */
+	nop
+	nop
+	nop
 	mr	r31,r3			/* save parameters */
 	mr	r30,r4
 	mr	r29,r5
===== arch/ppc/kernel/iSeries_head.S 1.9 vs edited =====
--- 1.9/arch/ppc/kernel/iSeries_head.S	Wed Nov  7 15:41:38 2001
+++ edited/arch/ppc/kernel/iSeries_head.S	Thu Mar 13 14:56:21 2003
@@ -80,6 +80,12 @@

 	.globl	__start
 __start:
+	/* To accomodate some SMP systems that overwrite the first few
+	 * locations before cpu 0 starts, the bootloader starts us at 0xc.
+	 */
+	nop
+	nop
+	nop
 	b	start_here


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Gabriel Paubert]

On Thu, Mar 13, 2003 at 03:38:53PM -0700, Dale Farnsworth wrote:
> -	li	r9,0x0000
> +	li	r9,0x000c
>  	mtlr	r9
>  	blr

Why not "ba 0x000c" instead? Just saves 8 bytes cheaply.

	Gabriel

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Patch boot/common/relocate.S to start kernel at 0x000c

[Dale Farnsworth]

On Fri, Mar 14, 2003 at 11:43:17AM +0000, Gabriel Paubert wrote:
> On Thu, Mar 13, 2003 at 03:38:53PM -0700, Dale Farnsworth wrote:
> > -	li	r9,0x0000
> > +	li	r9,0x000c
> >  	mtlr	r9
> >  	blr
>
> Why not "ba 0x000c" instead? Just saves 8 bytes cheaply.
>
> 	Gabriel

Looks good to me.
I didn't consider it because I've never seen ba before. :-)

-Dale

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/