Setting the physical RAM map

Setting the physical RAM map

[Adam Nielsen]

Hi all,

I'm attempting to port the Linux kernel to an NCD HMX, an R4600-based X-Terminal.

I've currently got it to the point where it will download the kernel and 
execute it, and start printing some messages out on the screen.

It gets as far as printing the physical RAM map and then crashes, but I'm not 
sure why:

   Determined physical RAM map:
    memory: 00800000 @ 40250000 (usable)
    memory: 00040000 @ 9fc00000 (ROM data)
   Wasting 8407552 bytes for tracking 262736 unused pages

   TLB refill exception PC = 40024094 address = 7FFFF000

The last message is from the boot monitor (the kernel is loaded at address 
0x40020000.)  I'm just guessing with the memory map, but I've tried lots of 
different values with the same result, and I'm fairly sure there is RAM mapped 
to the address I have used above (it's after the end of the kernel.)  At any 
rate the error message is from a completely different address, and it still 
happens if I flag that area as reserved memory in the RAM map.

Some of the MIPS devices don't seem to have code to create a memory map, so 
I'm wondering whether this is necessary or if there's a generic MIPS way to 
retrieve the current mapping.

Also, if anyone has any ideas what the kernel is trying to do accessing that 
invalid address I could use some hints!

Many thanks,
Adam.

Re: Setting the physical RAM map

[David VomLehn]

On Sun, Nov 29, 2009 at 12:38:55AM +1000, Adam Nielsen wrote:
> Hi all,
>
> I'm attempting to port the Linux kernel to an NCD HMX, an R4600-based X-Terminal.
...
> It gets as far as printing the physical RAM map and then crashes, but I'm 
> not sure why:
>
>   Determined physical RAM map:
>    memory: 00800000 @ 40250000 (usable)
>    memory: 00040000 @ 9fc00000 (ROM data)
>   Wasting 8407552 bytes for tracking 262736 unused pages

These are kernel messages, so you are getting into the kernel, but this
looks odd to me.  The MIPS processor needs memory mapped in the first
512 MiB to execute. This first 512 MiB will be mapped as cached memory
at virtual address 0x80000000 (known as kseg0) and as uncached memory at
virtual address 0x0a0000000 (known as kseg1). Perhaps you system is like
mine, where I have memory above 512 MiB aliased into the first 512 MiB
of physical space.

The second odd thing is the ROM data physical address. This is a good virtual
address but as a physical address it is not accessible by the MIPS processor
without mapping to with a TLB entry.

>   TLB refill exception PC = 40024094 address = 7FFFF000

This is not too surprising since the kernel is executing at an address
must have a TLB entry to be accessible.

> The last message is from the boot monitor (the kernel is loaded at 
> address 0x40020000.)  I'm just guessing with the memory map, but I've 
> tried lots of different values with the same result, and I'm fairly sure 
> there is RAM mapped to the address I have used above (it's after the end 
> of the kernel.)  At any rate the error message is from a completely 
> different address, and it still happens if I flag that area as reserved 
> memory in the RAM map.
>
> Some of the MIPS devices don't seem to have code to create a memory map, 
> so I'm wondering whether this is necessary or if there's a generic MIPS 
> way to retrieve the current mapping.

Some MIPS processors do not have an MMU, though you'd need to consult
the documentation to determine this. In your case, you're getting a
TLB refill exception, so you do have an MMU.

> Also, if anyone has any ideas what the kernel is trying to do accessing 
> that invalid address I could use some hints!

My guess, and since I don't have a lot of information this is a pretty
wild guess, is that you are actually running your kernel at 0x40020000,
and that the reason you can do this is that the bootloader has left the
TLBs in such a state that the first few pages are mapped. Then, when it
leaves the mapped area, it dies with the TLB refill exception.

Another possibility is that you actually start off at reasonable 0x8xxxxxxx
address, but then use a pointer that takes you off to 0x40024094.

One suggestion is to look at the code at 0x40024094 to see what it's doing.
Then, use a JTAG debugger or add printk() statements to see exactly what's
happening.

-- 
David VL

Re: Setting the physical RAM map

[Geert Uytterhoeven]

On Mon, Nov 30, 2009 at 20:00, David VomLehn <dvomlehn@cisco.com> wrote:
> On Sun, Nov 29, 2009 at 12:38:55AM +1000, Adam Nielsen wrote:
>> I'm attempting to port the Linux kernel to an NCD HMX, an R4600-based X-Terminal.
> ...
>> It gets as far as printing the physical RAM map and then crashes, but I'm
>> not sure why:
>>
>>   Determined physical RAM map:
>>    memory: 00800000 @ 40250000 (usable)
>>    memory: 00040000 @ 9fc00000 (ROM data)
>>   Wasting 8407552 bytes for tracking 262736 unused pages
>
> These are kernel messages, so you are getting into the kernel, but this
> looks odd to me.  The MIPS processor needs memory mapped in the first
> 512 MiB to execute. This first 512 MiB will be mapped as cached memory
> at virtual address 0x80000000 (known as kseg0) and as uncached memory at
> virtual address 0x0a0000000 (known as kseg1). Perhaps you system is like
> mine, where I have memory above 512 MiB aliased into the first 512 MiB
> of physical space.
>
> The second odd thing is the ROM data physical address. This is a good virtual
> address but as a physical address it is not accessible by the MIPS processor
> without mapping to with a TLB entry.

9fc00000 is kseg0. kseg0 has 512 MiB, so that's not only 0x8???????, but also
0x9???????.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Setting the physical RAM map

[Ralf Baechle]

On Sun, Nov 29, 2009 at 12:38:55AM +1000, Adam Nielsen wrote:

> I'm attempting to port the Linux kernel to an NCD HMX, an R4600-based X-Terminal.
> 
> I've currently got it to the point where it will download the kernel
> and execute it, and start printing some messages out on the screen.
> 
> It gets as far as printing the physical RAM map and then crashes,
> but I'm not sure why:
> 
>   Determined physical RAM map:
>    memory: 00800000 @ 40250000 (usable)
>    memory: 00040000 @ 9fc00000 (ROM data)
>   Wasting 8407552 bytes for tracking 262736 unused pages
> 
>   TLB refill exception PC = 40024094 address = 7FFFF000
> 
> The last message is from the boot monitor (the kernel is loaded at
> address 0x40020000.)  I'm just guessing with the memory map, but
> I've tried lots of different values with the same result, and I'm
> fairly sure there is RAM mapped to the address I have used above
> (it's after the end of the kernel.)  At any rate the error message
> is from a completely different address, and it still happens if I
> flag that area as reserved memory in the RAM map.

Are you sure it's a R4600, not R4640 or R4650?

It's like a decade that I last read up on these but afair they have a
fixed mapping starting at 0x40000000.  It would make perfect sense to
use such a CPU in an X terminal.

  Ralf

Re: Setting the physical RAM map

[Adam Nielsen]

>>    Determined physical RAM map:
>>     memory: 00800000 @ 40250000 (usable)
>>     memory: 00040000 @ 9fc00000 (ROM data)
>>    Wasting 8407552 bytes for tracking 262736 unused pages
>
> These are kernel messages, so you are getting into the kernel, but this
> looks odd to me.  The MIPS processor needs memory mapped in the first
> 512 MiB to execute. This first 512 MiB will be mapped as cached memory
> at virtual address 0x80000000 (known as kseg0) and as uncached memory at
> virtual address 0x0a0000000 (known as kseg1). Perhaps you system is like
> mine, where I have memory above 512 MiB aliased into the first 512 MiB
> of physical space.
>
> The second odd thing is the ROM data physical address. This is a good virtual
> address but as a physical address it is not accessible by the MIPS processor
> without mapping to with a TLB entry.

Are these supposed to be physical addresses?  Maybe that's where I'm going 
wrong.  I only know the virtual addresses from some experimentation, but I've 
got no idea how to find out the physical addresses.

The system won't accept the kernel unless its virtual load address is 
0x40020000.  If I use the 'sm' (show memory) command in the boot loader it 
tells me there is 16MB of RAM starting at 0x20000000 and 8MB starting at 
0x21000000, so but if memory is supposed to start at 0x0 then those don't look 
like physical addresses either.

>>    TLB refill exception PC = 40024094 address = 7FFFF000
>
> This is not too surprising since the kernel is executing at an address
> must have a TLB entry to be accessible.

 From my other experiments here, 0x40024094 is fine, the problem is that the 
kernel is trying to access the unmapped address at 0x7FFFF000 and I'm not sure 
why.  It looks like an unusual address - is it trying to trigger an exception 
to test whether some exception handler has been installed correctly?

> My guess, and since I don't have a lot of information this is a pretty
> wild guess, is that you are actually running your kernel at 0x40020000,
> and that the reason you can do this is that the bootloader has left the
> TLBs in such a state that the first few pages are mapped. Then, when it
> leaves the mapped area, it dies with the TLB refill exception.
>
> Another possibility is that you actually start off at reasonable 0x8xxxxxxx
> address, but then use a pointer that takes you off to 0x40024094.

I'm pretty sure execution starts at 0x40020000.  With some testing code I have 
successfully written at 0x40040000 so it looks like that's where the boot 
loader has mapped a fair chunk of memory.  0x40024094 shouldn't be a problem, 
but 0x7FFFF000 probably would be.

> One suggestion is to look at the code at 0x40024094 to see what it's doing.
> Then, use a JTAG debugger or add printk() statements to see exactly what's
> happening.

I think that's probably the best bet.  Unfortunately figuring out where that 
address ends up in the C source is a little tricky for me, and I'm still a bit 
slow at reading MIPS assembly!

Thanks for your helpful reply.

Cheers,
Adam.

Re: Setting the physical RAM map

[Adam Nielsen]

> Are you sure it's a R4600, not R4640 or R4650?
>
> It's like a decade that I last read up on these but afair they have a
> fixed mapping starting at 0x40000000.  It would make perfect sense to
> use such a CPU in an X terminal.

Hmm, I'm not sure.  I'd have to peel off the heatsink to be sure I guess.  The 
very first kernel messages print this:

   CPU revision is: 00002020 (R4600)
   FPU revision is: 00002020

So I guess these values would be more specific if the CPU was indeed one of 
those revisions.

Cheers,
Adam.

Re: Setting the physical RAM map

[Ralf Baechle]

On Tue, Dec 01, 2009 at 08:38:39AM +1000, Adam Nielsen wrote:

> >It's like a decade that I last read up on these but afair they have a
> >fixed mapping starting at 0x40000000.  It would make perfect sense to
> >use such a CPU in an X terminal.
> 
> Hmm, I'm not sure.  I'd have to peel off the heatsink to be sure I
> guess.  The very first kernel messages print this:
> 
>   CPU revision is: 00002020 (R4600)
>   FPU revision is: 00002020
> 
> So I guess these values would be more specific if the CPU was indeed
> one of those revisions.

No need to scratch off things.  The numbers above clearly identify an
R4600 v2.0.  R4640 and R4650 would return 0x000022xx where xx would be
the processor revision number.  Both are special embedded versions of the
R4600 without MMU.

  Ralf

Re: Setting the physical RAM map

[Adam Nielsen]

Hi all,

I've done some more debugging and tracked down the problem a little further. 
I've also fixed up (I think) the memory mappings, from looking at the setup 
code from some of the other MIPS machines:

Determined physical RAM map:
  memory: 01800000 @ 00000000 (usable)
  memory: 00040000 @ 1fc00000 (ROM data)

>>    TLB refill exception PC = 40024094 address = 7FFFF000
>
> This is not too surprising since the kernel is executing at an address
> must have a TLB entry to be accessible.

The boot loader refuses to execute the ELF image unless the load address is 
set to 0x40020000 ("Load address out of range"), so it appears the kernel is 
being executed at this address.  The problem seems to be that there is nothing 
mapped at 0x7FFFF000.

I've tracked the code that accesses this memory address to the 
init_bootmem_core() function in mm/bootmem.c line ~109:

   memset(bdata->node_bootmem_map, 0xff, mapsize);

This is being executed as:

   memset(0x7ffff000, 0xff, 768);

Which is where the problem is coming from.  Working backwards, I have narrowed 
it down to arch/mips/kernel/setup.c line ~293.  This is a loop which does some 
calculations with memory (not sure exactly what) but the "mapstart" variable 
is initialised to ~0UL, and it never gets updated before being passed through 
to eventually the memset() line above.

The problem seems to be inside the loop.  These lines:

   if (end <= reserved_end)
     continue;

Cause the loop to break out *before* setting mapstart, and since there is only 
one RAM element in the array the loop does not run again.  It seems that the 
end of the kernel (reserved_end) is so big (it'll be 0x40020000 + size of 
kernel) that it sits way after the end of the RAM mapping (0x01800000).

I'm not sure how to solve this issue, and I'm still a bit confused about MIPS 
memory mapping (does the TLB mean that 0x40020000 could be mapped anywhere in 
memory?  Will this break things when Linux starts reprogramming it?  Or does 
Linux leave the TLB alone?)  I've tried changing the memory from appearing at 
offset 0 to offset 0x40020000 but it didn't change anything (just a message 
about 8MB wasted on tracking unused pages.)

Any pointers would be greatly appreciated!

Many thanks,
Adam.