problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Thomas Bogendoerfer]

I tried to get a working 64bit kernel for SNI RM. Most of things
to fix were quite obvious, but there is one thing, which I haven't
understood yet.

The firmware is only able to boot ELF32 images, which mean I need to
use BOOT_ELF32.

RM machines have 256MB memory mapped to KSEG0, anything else is outside.
To me that would mean I need to use the default spaces from
mach-generic/spaces.h. A kernel built that way will hang after calling
schedule() in rest_init() (init/main.c). Has anybody seen this
as well ?

Before digging into schedule() I decided to try mach-ip22/spaces.h
and limit the installed memory to 256MB. This kernel boots and
runs fine.

Then I booted that kernel with all memory (512MB) and it died, when
init had troubles mapping libc. That result didn't surprise me
that much, since the kernel probably tried to map memory > 256MB
via CKSEG0, which won't work. Correct ?

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Ralf Baechle]

On Sun, Nov 11, 2007 at 03:33:02PM +0100, Thomas Bogendoerfer wrote:

> I tried to get a working 64bit kernel for SNI RM. Most of things
> to fix were quite obvious, but there is one thing, which I haven't
> understood yet.
> 
> The firmware is only able to boot ELF32 images, which mean I need to
> use BOOT_ELF32.
> 
> RM machines have 256MB memory mapped to KSEG0, anything else is outside.
> To me that would mean I need to use the default spaces from
> mach-generic/spaces.h. A kernel built that way will hang after calling
> schedule() in rest_init() (init/main.c). Has anybody seen this
> as well ?

No.

schedule() doesn't directly depend on very much else working so I get the
feeling that your problem may be quite far away.

> Before digging into schedule() I decided to try mach-ip22/spaces.h
> and limit the installed memory to 256MB. This kernel boots and
> runs fine.
> 
> Then I booted that kernel with all memory (512MB) and it died, when
> init had troubles mapping libc. That result didn't surprise me
> that much, since the kernel probably tried to map memory > 256MB
> via CKSEG0, which won't work. Correct ?

You should keep all memory mapped in the same kernel segment.  Since
you have too much to do that in CKSEG0 it will have to be XKPHYS which
means the generic spaces.h which will be used when you have none in
in mach-rm200/space.h.

I suspect what happens is the kernel is trying to access memory at above
phys. 512MB.  In your setup it would compute a CKSEG1 address for that
and stomp over something it better shouldn't ...

  Ralf

Re: problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Thomas Bogendoerfer]

On Sun, Nov 11, 2007 at 09:31:28PM +0000, Ralf Baechle wrote:
> On Sun, Nov 11, 2007 at 03:33:02PM +0100, Thomas Bogendoerfer wrote:
> 
> > I tried to get a working 64bit kernel for SNI RM. Most of things
> > to fix were quite obvious, but there is one thing, which I haven't
> > understood yet.
> > 
> > The firmware is only able to boot ELF32 images, which mean I need to
> > use BOOT_ELF32.
> > 
> > RM machines have 256MB memory mapped to KSEG0, anything else is outside.
> > To me that would mean I need to use the default spaces from
> > mach-generic/spaces.h. A kernel built that way will hang after calling
> > schedule() in rest_init() (init/main.c). Has anybody seen this
> > as well ?
> 
> No.
> 
> schedule() doesn't directly depend on very much else working so I get the
> feeling that your problem may be quite far away.

well it depends on getting a kernel thread started. And I guess there
lies the problem.

Maybe I need to ask more precisely what I want to know: Should a
kernel linked to CSKEG0 with a CAC_BASE 0x980000000000 work or is 
this setup "wrong" ? If the answer is yes, I'd say there is still
a bug left. If no how do we solve the issue of having a 32bit firmware
and a kernel linked for XPHYS (using some sort of bootloader is not
want I want to have, if it's avoidable) ?

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Ralf Baechle]

On Mon, Nov 12, 2007 at 09:32:42AM +0100, Thomas Bogendoerfer wrote:

> > schedule() doesn't directly depend on very much else working so I get the
> > feeling that your problem may be quite far away.
> 
> well it depends on getting a kernel thread started. And I guess there
> lies the problem.
> 
> Maybe I need to ask more precisely what I want to know: Should a
> kernel linked to CSKEG0 with a CAC_BASE 0x980000000000 work or is 
> this setup "wrong" ? If the answer is yes, I'd say there is still
> a bug left. If no how do we solve the issue of having a 32bit firmware
> and a kernel linked for XPHYS (using some sort of bootloader is not
> want I want to have, if it's avoidable) ?

Typically systems try to put the kernel into CKSEG0 as long as the given
platform happens to have memory mapped there.  Using CKSEG0 allows using
a shorter sequence to load the address of an in-kernel symbol resulting
in a much smaller kernel binary.

One implication of this is that kernel modules will have to be allocated
to CKSEG2/3 because they're built using the same code model.  Modules
are allocated using vmalloc so the vmalloc space needs to be in
CKSEG2/3.  Most people don't care if that space is restricted to just
one gig.

But even if you get that wrong the expected failure mode is different ...

  Ralf

Re: problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Thomas Bogendoerfer]

On Mon, Nov 12, 2007 at 10:44:23AM +0000, Ralf Baechle wrote:
> But even if you get that wrong the expected failure mode is different ...

Ralf and me had an debug session on IRC and I finally figured out
what caused the problem: CONFIG_EARLY_PRINTK via prom calls.

I simply used call_o32.S from the decstation part and missed the
fact, that it simply uses the normal kernel stack when calling
firmware. This works quite good until the first kernel thread
gets scheduled, which has a kernel stack via a CAC_BASE address.
So after switching stack the next call to prom_putchar() killed the
machine. Simply disabling EARLY_PRINTK gives me a working 64bit
kernel, which sees the whole 512MB RAM :-)

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: problem with 64bit kernel, BOOT_ELF32 and memory outside CKSEG0

[Maciej W. Rozycki]

On Mon, 12 Nov 2007, Thomas Bogendoerfer wrote:

> I simply used call_o32.S from the decstation part and missed the
> fact, that it simply uses the normal kernel stack when calling
> firmware. This works quite good until the first kernel thread
> gets scheduled, which has a kernel stack via a CAC_BASE address.

 You could do stack switching in call_o32() -- I just figured there was no 
point in adding this complication as the DECstation always runs from KSEG0 
-- it has the maximum of 480MB of RAM mapped linearly starting from 0.

  Maciej