Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Christer Weinigel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]


alan.corp.packetengines.com wrote:

>I tried using the MD_CAM and MD_RAMn registers to debug this.  These seem to
>read back the values I have stored via the MD_RPN, MD_TWC and MD_RPN register
>EXCEPT that when I read back I get a value for the MD_CAM.PS field of 110
>rather than the 111 which is the documented 8MB page size value.  The doc even
>says that values other than those listed (000, 001, 011, 111) are "reserved".
>The value I am setting in the MD_TWC.PS field (which is oddly a two bit
>encoding for page size) is 11, which is documented as the 8MB page
>size value.

This is simple :-)  Or rather, I was tearing my hair out last week
trying to figure out exactly the same thing.  Set the 4/16k bit in
MD_RPN to 1 and it'll work.  Apparently, the two page size bits in
MD_TWC and the bit från MD_RPN together form the real page size.  The
documentation is a bit misleading here.

Now I'd just like to know why I can't get 8 MB pages working with the
Linux page tables.  The performance boost from putting the Linux
kernel in an 8 MB page on a MPC850 is rather impressive, most lmbench
system call benchmarks went twice as fast.

  /Christer

--
Christer Weinigel		Cendio Systems AB
Email:	wingel@cendio.se	Teknikringen 8
Phone:	+46-13-21 46 00		583 30  Linköping
Fax:	+46-13-21 47 00		Sweden
[please note that Signum Support has changed its name to Cendio Systems]

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Dan Malek]

Christer Weinigel wrote:

> ........  Set the 4/16k bit in
> MD_RPN to 1 and it'll work.  Apparently, the two page size bits in
> MD_TWC and the bit från MD_RPN together form the real page size.  The
> documentation is a bit misleading here.

That is the proper thing to do, but not exactly the reason.  You
should also set MD_CTR.TWAM = 1, since the problem you are solving
is telling the MMU not to use 4K/1K subpages.


> Now I'd just like to know why I can't get 8 MB pages working with the
> Linux page tables.


How much software are you trying to modify in the generic Linux MM
functions?  If none, it isn't likely to work except in one condition.
You will have to wire the pages into permanent TLB entries, and
move the kernel VM map around to ensure dynamically allocated
pages don't overlap the 8M "pages".  This is the first and easiest
to do, but......


> ....  The performance boost from putting the Linux
> kernel in an 8 MB page on a MPC850 is rather impressive, most lmbench
> system call benchmarks went twice as fast.


....lmbench is not necessary representative of embedded applications.
Performing the above modification to some of the embedded applications
I have developed would reduce the performance because the number of
TLB entries in the 850 or 823 is rather small.

As I have said in previous messages, you can find quick little
hacks like this that may help a specific application, but they
are not generically beneficial.  The real solution is to significantly
modify the Linux VM macros and functions to allow a better formatted
page table and reduce the amount of instructions in the TLB handler.
Further, we could leverage a variety of page sizes simultaneously.
This has been on my list for a long time, but isn't a trivial
weekend project (especially with the VM changes in 2.3.xx).


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Alan Mimms]

Thanks, Dan, for the additional suggestion.

I would like to suggest that using one or two 512MB pages locked into the
TLB's top two slots using the "RSVDn" flag in the Mx_CTL register might be the
right way to go.  Unless you can physically see the RAM present in the 8MB
single TLB entry's mapping in two separate physical addresses (more than 8MB
apart), you'll basically have to waste the memory beyond the top of the
kernel's requirements with this scheme.  Granularity of 512MB would be a good
deal closer and then you can effectively reserve 1MB for the kernel (close to
its actual size) pretty easily.

Or am I misunderstanding the issue somehow?

a

 On Tue, 18 Jan 2000, Dan Malek wrote:
> Christer Weinigel wrote:
>
> > ........  Set the 4/16k bit in
> > MD_RPN to 1 and it'll work.  Apparently, the two page size bits in
> > MD_TWC and the bit från MD_RPN together form the real page size.  The
> > documentation is a bit misleading here.
>
> That is the proper thing to do, but not exactly the reason.  You
> should also set MD_CTR.TWAM = 1, since the problem you are solving
> is telling the MMU not to use 4K/1K subpages.
>
>
> > Now I'd just like to know why I can't get 8 MB pages working with the
> > Linux page tables.
>
>
> How much software are you trying to modify in the generic Linux MM
> functions?  If none, it isn't likely to work except in one condition.
> You will have to wire the pages into permanent TLB entries, and
> move the kernel VM map around to ensure dynamically allocated
> pages don't overlap the 8M "pages".  This is the first and easiest
> to do, but......
>
>
> > ....  The performance boost from putting the Linux
> > kernel in an 8 MB page on a MPC850 is rather impressive, most lmbench
> > system call benchmarks went twice as fast.
>
>
> ....lmbench is not necessary representative of embedded applications.
> PPerforming the above modification to some of the embedded applications
> I have developed would reduce the performance because the number of
> TLB entries in the 850 or 823 is rather small.
>
> As I have said in previous messages, you can find quick little
> hacks like this that may help a specific application, but they
> are not generically beneficial.  The real solution is to significantly
> modify the Linux VM macros and functions to allow a better formatted
> page table and reduce the amount of instructions in the TLB handler.
> Further, we could leverage a variety of page sizes simultaneously.
> This has been on my list for a long time, but isn't a trivial
> weekend project (especially with the VM changes in 2.3.xx).
>
>
> 	-- Dan
--
Alan Mimms     Packet Engines, Inc.     Spokane, Washington [99214-0497]
  USA, Earth, Sol, Milky Way, The Local Group, Virgo Supercluster, U0
Despite the cost of living, have you noticed how popular it remains?
  -- Steven Wright?

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Dan Malek]

Alan Mimms wrote:


> I would like to suggest that using one or two 512MB pages


That isn't what I would do.  You can map the entire kernel
instruction space with a single 8M entry.  You can map the
entire static data space with another 8M entry.  You can
map frequently used I/O with another 8M data entry.

You have to move the VM map so dynamically allocated data
doesn't overlap the 8M page.  You have to make sure dynamically
loaded modules don't overlap the 8M instruction map.  Or, make
sure they overlap entirely (either way is fine, just don't mix
it up).  You can modify the ioremap() functions to operate more
like they do on the processors with BATs.

All of these are likely to be useful in most applications.  They
don't require any Linux VM or TLB software changes.  All you
need to change are some #define configuration vaules in a couple
of VM/MMU header files and slightly change the MMU initialization.

Maybe I'll do this later today.....I have been thinking about it
long enough.



	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Christer Weinigel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2403 bytes --]


Dan Malek wrote:

> How much software are you trying to modify in the generic Linux MM
> functions?  If none, it isn't likely to work except in one condition.
> You will have to wire the pages into permanent TLB entries, and
> move the kernel VM map around to ensure dynamically allocated
> pages don't overlap the 8M "pages".  This is the first and easiest
> to do, but......

Nope, that's not at all what I'm doing.  What I did was to modify the
data (and instruction) TLB miss handlers, first to add some TLB
statistics, and later to do the 8 MB pages hack, like this:

    dtlb_misses++;		/* update statistics */

#ifdef UGLY_HACK
    addr = MD_EPN & 0xfffff000;
    if (addr >= 0xc0000000 && addr < 0xc0000000 + memory_size)
    {
        MD_TWC = MD_PS8MEG | MD_SVALID;
        MD_RPN = addr | 0xf0 | MD_PS4K_16K | MD_SVALID;
        return;
    }
#endif

    /* normal TLB handler */
    return;

Which means that a normal user application will be able to use all 8
TLB entries, but upon entering the kernel there will usually be two
page faults, one to load one ITLB entry and later one DTLB entry.
Yes, I do know that this slows down each and every TLB miss, but it
was just a quick hack to see if it was worth doing at all.  The
results were rather promising, lmbench went much faster, and just
counting the number of TLB misses from boot until the kernel has been
able to mount an NFS server and show a bash prompt gave this result:

		   misses
		ITLB	DTLB
Unmodified	120k	260k
8 MB hack	15k	35k

What I've been trying to do next is to put the 8 MB page information
in the pgd/pte entries, because I don't want the speed loss coming
from the compare in a time critical code path.  I have read the
comments in the source about pgd entries being used as pointers, so to
begin with, I just tried to change the mapin_ram function to:

    map_page(&init_task, v, p, f | 8);

so that the kernel view of the RAM would use 16 kB pages instead of 4
kB pages.  This fails for some reason and that's what I've been trying
to fix now.  That it what I'd like to get working first of all.

  /Christer

--
Christer Weinigel		Cendio Systems AB
Email:	wingel@cendio.se	Teknikringen 8
Phone:	+46-13-21 46 00		583 30  Linköping
Fax:	+46-13-21 47 00		Sweden
[please note that Signum Support has changed its name to Cendio Systems]

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

WARNING OFF TOPIC SLIGHTLY: 8xx MMU w/8MB pages

[Alan Mimms]

Hello all,

If you are only interested in Linux on 8xx PPC, please ignore this message.  I
am having a problem building up MMU tables for 8MB pages on a NON-Linux system
running on an 850.  I hope someone on this list can help (PLEASE!).

The system works fine but is "slow".  I look into this and find that my DTLB
miss exception handler is running a LOT (50k exceptions/sec!).  Seems the TLB
is never matching.  I have the MMU set up for all mappings to be 8MB pages to
map 32MB of RAM and a few I/O devices.  There are slightly more than 8 PTEs
that are in use in the system so I can't statically load up the TLB and get on
with life - I have to handle the DTLB misses.  But I don't use the 8xx
documentation's specified page table structure since I have a very flat
logical==physical mapping.  I'm simply using the MMU to flag cacheable vs
noncacheable spaces, really.

The MD_CTR is set to 0.  I don't have TWAM or PPCS on.  This is a wide-open no
protection kind of system, so I can get away with very little actual MMU bells
and whistles.

I tried using the MD_CAM and MD_RAMn registers to debug this.  These seem to
read back the values I have stored via the MD_RPN, MD_TWC and MD_RPN register
EXCEPT that when I read back I get a value for the MD_CAM.PS field of 110
rather than the 111 which is the documented 8MB page size value.  The doc even
says that values other than those listed (000, 001, 011, 111) are "reserved". 
The value I am setting in the MD_TWC.PS field (which is oddly a two bit
encoding for page size) is 11, which is documented as the 8MB page size value.

The TLB entries are all being written with ASID of zero, which is the value in
CASID so these can't be the reason for the mismatch.

Here's a sample of what I am doing:

M$reg md_ctr=700
!!! look at TLB entry #7

M$reg md_epn=00200 md_twc=0d md_rpn=009f5
!!! write entry with
!!!  MD_EPN: EPN=0 EV=1 ASID=0
!!!  MD_TWC: APG=0 G=0 PS=11 (8MB) V=1
!!!  MD_RPN: RPN=0 PP={supvR/W, PowerPC, Change, all subpages valid},
!!!          SPS=0 (wrong?) SH=1 CI=0 V=1

M$reg md_ctr=700
!!! Reset to look at TLB entry #7

$cam
  reg md_ctr=00000700
  reg md_dbcam=00000fd0
  reg md_dbram0=00000de0
  reg md_dbram1=00007fff

!!! This shows I am looking at entry #7,
!!! and the MD_CAM: EPN=0 SPVF=1111 PS=110 (!) SH=1 ASID=0
!!! (MD_CAM is called MD_DBCAM for some reason on the HP debugging box I use).

Note that the above MD_CAM shows a PS of 110 which is RESERVED according to the
MPC850 documentation.

What am I doing wrong?  Is there more info one would need to be able to answer
this question?

PLEASE HELP.

Thanks very much.
a
 -- 
Alan Mimms     Packet Engines, Inc.     Spokane, Washington [99214-0497]
  USA, Earth, Sol, Milky Way, The Local Group, Virgo Supercluster, U0
Despite the cost of living, have you noticed how popular it remains?
  -- Steven Wright?

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/