Using more than 256 MB of memory on SB1250 in 32-bit mode, revisited

["Matthew Starzewski" <mstarzewski@xes-inc.com>]

[-- Attachment #1: Type: text/plain, Size: 6020 bytes --]

I've tried to enable HIGHMEM to access all 512MB of
SDRAM on a BCM1125 based board as per this previous
thread:

Using more than 256 MB of memory on SB1250 in 32-bit mode :
http://www.spinics.net/lists/mips/msg14396.html
BCM1125 Board: XPedite3000 PrPMC
http://www.xes-inc.com/Products/XPedite/XPedite3000/XPedite3000.html

In MIPS32 mode, the memory map comes out to the following:

Physical Memory Map:
0x00000000 - 0x0FFFFFFF   256MB
0x80000000 - 0x8FFFFFFF   256MB
Virtual Memory Map:
0x80000000 - 0x8FFFFFFF   256MB
<<<< INACCESSIBLE >>>> 256MB

My goal in enabling HIGHMEM was to get at the upper 256MB
much as described here.  Access to the upper 256MB through
HIGHMEM may incur a performance hit, but it's certainly better
going without.

http://home.earthlink.net/~jknapka/linux-mm/vminit.html#PAGING_INIT

However, what I get is a stall as per the log pasted below.  Enabling
CONFIG_64BIT_PHYS_ADDR does not make a difference.  Results
were cross-verified between CFE and another bootloader.

When I hand the physical memory described above off to add_memory_region,
I noticed a few odd things. One thing that looks suspicious is that the variable
void *high_memory ends up being set to 0xa0000000, or right at KSEG1,
in mm/init.c.

Also, num_physpages became huge, 0x90000, because the init code in
kernel/setup.c and mm/init.c want a page for every memory location, even
highmem.  Is this appropriate when the memory is not directly accessible
via __va and virt_to_phys?  

This may be an ancillary effect of what's mentioned above, but when num_physpages
grows in size, nr_free_pages doesn't track with it, so in void vfs_caches_init(unsigned long
mempages) and the like, you get a horrible underflow condition:

/* code */
        printk("MJS - nr_free_pages():0x%X\n", nr_free_pages());
        printk("MJS - OLD mempages:0x%X\n", mempages);
        reserve = (mempages - nr_free_pages()) * 3/2;
        mempages -= reserve;
        printk("MJS - NEW reserve:0x%X mempages:0x%X\n",
               reserve, mempages);

/* printout */
MJS - nr_free_pages():0x1E9A0
MJS - OLD mempages:0x90000
MJS - NEW reserve:0xAA190 mempages:0xFFFE5E70


Let me know what you think of this issue.  In anticipation of the
"Why not use MIPS64 build?" question, we'd prefer to and will, but the 
MIPS64 build has underperformed or had bugs (SATA seek time,
networking signals, etc) that MIPS32 doesn't.  For these issues
or any case where the MIPS64 build isn't there yet, it makes
sense to have the MIPS32 path open.

Regards,
Matt


============= Kernel Boot Log ==================

CFE version 1.0.37 for BCM91125E (64bit,SP,BE)
Build Date: Mon Jun 28 18:33:29 CDT 2004 (mstarzewski@lsys1)
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.

Initializing Arena.
Initializing Devices.
BCM91125E board revision 1
Config switch: 0
CPU: 1125H A2
L2 Cache: 256KB
SysCfg: 00800000004A0600 [PLL_DIV: 12, IOB0_DIV: CPUCLK/4, IOB1_DIV: CPUCLK/3]
CPU type 0x40103: 600MHz
Total memory: 0x20000000 bytes (512MB)

Total memory used by CFE:  0x8FE8AFC0 - 0x90000000 (1527872)
Initialized Data:          0x8FE8AFC0 - 0x8FE94D40 (40320)
BSS Area:                  0x8FE94D40 - 0x8FE95430 (1776)
Local Heap:                0x8FE95430 - 0x8FF95430 (1048576)
Stack Area:                0x8FF95430 - 0x8FF97430 (8192)
Text (code) segment:       0x8FF97440 - 0x8FFFFFB8 (428920)
Boot area (physical):      0x0FE49000 - 0x0FE89000
Relocation Factor:         I:F0397440 - D:0DF8AFC0

CFE> ifconfig eth0 -addr=10.52.33.67 -mask=255.255.0.0;boot -elf 10.52.0.4:/home/mstarzewski/li
nux/kernels/mips26_works_cfe/vmlinux
eth0: Link speed: 100BaseT FDX
Device eth0:  hwaddr 40-00-10-06-40-00, ipaddr 10.52.33.67, mask 255.255.0.0
        gateway not set, nameserver not set
Loader:elf Filesys:tftp Dev:eth0 File:10.52.0.4:/home/mstarzewski/linux/kernels/mips26_works_cf
e/vmlinux Options:(null)
Loading: 0xffffffff80100000/2558312 0xffffffff80370968/146032 Entry at 0xffffffff8032f018
Variable Name        Value
-------------------- --------------------------------------------------
BOOT_CONSOLE         uart0
CPU_TYPE             1125H
CPU_REVISION         A2
CPU_NUM_CORES        1
CPU_SPEED            600
CFE_VERSION          1.0.37
CFE_BOARDNAME        BCM91125E
CFE_MEMORYSIZE       512
NET_DEVICE           eth0
NET_IPADDR           10.52.33.67
NET_NETMASK          255.255.0.0
NET_GATEWAY          0.0.0.0
NET_NAMESERVER       0.0.0.0
BOOT_DEVICE          eth0
BOOT_FILE            10.52.0.4:/home/mstarzewski/linux/kernels/mips26_works_cfe/vmlinux
DELETING BOOT_CONSOLE
DELETING CPU_REVISION
DELETING CPU_SPEED
DELETING CFE_BOARDNAME
DELETING NET_DEVICE
DELETING NET_NETMASK
DELETING NET_NAMESERVER
DELETING BOOT_FILE
Variable Name        Value
-------------------- --------------------------------------------------
Closing network.
Starting program at 0xffffffff8032f018
CFE addr:0x000000000FE8A000 size:0x0000000000000000, type:0
CFE addr:0x0000000010000000 size:0x0000000000000000, type:-2147483648
Broadcom SiByte BCM1125H A2 @ 600 MHz (SB1 rev 3)
Board type: SiByte BCM91250A (SWARM)
Linux version 2.6.6-rc3 (mstarzewski@lsys1) (gcc version 3.2.3 with SiByte modifications) #17 S
MP Thu Dec 9 11:08:23 CST 2004
CPU revision is: 00040103
FPU revision is: 000f0103
This kernel optimized for board runs with CFE
Determined physical RAM map:
 memory: 0fe89e00 @ 00000000 (usable)
 memory: 0ffffe00 @ 80000000 (usable)
1791MB HIGHMEM available.
On node 0 totalpages: 589823
  DMA zone: 131072 pages, LIFO batch:16
  Normal zone: 0 pages, LIFO batch:1
  HighMem zone: 458751 pages, LIFO batch:16
Built 1 zonelists
Kernel command line: root=/dev/nfs ip=auto rw
PID hash table entries: 4096 (order 12: 32768 bytes)
Memory: 495040k/260644k available (1850k kernel code, 27020k reserved, 382k data, 264k init, 26
2140k highmem)
Calibrating delay loop... 399.36 BogoMIPS

<< STALL >>

[-- Attachment #2: Type: text/html, Size: 12088 bytes --]