Re: CVS Update@-mips.org: linux

Re: CVS Update@-mips.org: linux

[Ralf Baechle]

On Tue, Oct 14, 2003 at 11:47:59AM +0200, Maciej W. Rozycki wrote:

> > Log message:
> > 	Config bits of support for TLB page sizes other than 4k.  So not
> > 	functional yet but this is a fairly large project and everybody should
> > 	finally get rid of the assumption that PAGE_SIZE is always 4k.
> 
>  I've already thought we might unconditionally switch to a larger page
> size, e.g. 16kB, for the 64-bit kernel -- I suppose everything that
> supports 64-bit operation also supports larger page sizes, doesn't it?

True.  At 16k pagesize we can also start harvesting other advantages, for
example we'd then know that cpu_has_dc_aliases is never true which will
make the compiler simply throw away large parts of the cache code.

As the result of living in an alias-free universe we could use the nice
recursive page fault technique from very old Linux kernels again,
something that allowed blindigly fast TLB refill handlers.

An additional bonus is the increased coverage of a pagetables with 16k
page size.  A 16k page has space for 2048 pointers.  So in case of a
two level pagetable tree that makes the total capacity 16k*2k*2k = 64GB.
So that means one level of the pagetable tree less than we currently
have, yet enough capacity for everything but the largest jobs.

Still want more?  A 3 level tree would then cover 128TB of virtual
address space already exceedin the hardware limits of all processors but
the R8000.

64k pagesize stretches the limits even further.   Here a two level
pagetable tree would cover 4TB, 3-level could cover 32PB exceeding
the capacity of every MIPS processor ever made - and probably sufficient
for the coming decade :-)

  Ralf

Re: CVS Update@-mips.org: linux

[Maciej W. Rozycki]

On Tue, 14 Oct 2003, Ralf Baechle wrote:

> Still want more?  A 3 level tree would then cover 128TB of virtual
> address space already exceedin the hardware limits of all processors but
> the R8000.

 Well, the MIPS64 ISA spec allows up to 8EB of user memory to be supported
by an implementation, IIRC; probably nothing supports that much yet,
though. ;-)  BTW, is an R8000 spec available online anywhere?

> 64k pagesize stretches the limits even further.   Here a two level
> pagetable tree would cover 4TB, 3-level could cover 32PB exceeding
> the capacity of every MIPS processor ever made - and probably sufficient
> for the coming decade :-)

 Further increasing of the page size should result in better performance
due to fewer TLB misses and reduce the memory footprint of page tables,
but the drawback is more memory is wasted for maps.  Whether the end
result is a gain or a loss depends on the actual application of a system,
so I guess we should either leave the size configurable (with a sane
default for those who might have troubles judging what would suit them
best) or only decide on a given size after lots of benchmarking.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

MIPS addressing limits, Was: Re: CVS Update@-mips.org: linux

[Ralf Baechle]

On Wed, Oct 15, 2003 at 04:23:06PM +0200, Maciej W. Rozycki wrote:

> > Still want more?  A 3 level tree would then cover 128TB of virtual
> > address space already exceedin the hardware limits of all processors but
> > the R8000.
> 
>  Well, the MIPS64 ISA spec allows up to 8EB of user memory to be supported
> by an implementation, IIRC; probably nothing supports that much yet,
> though. ;-)  BTW, is an R8000 spec available online anywhere?

There used to be a few papers published by SGI online and various other
bits of information I found through google.

(I happen to have a paper copy of the R8000 manual but since the responsible
people still haven't informed me if I can legally use it, this book is
closed and will stay closed until this happens - if ever ...  Pitty, I
still receive mails from various R8000 users ...)

> > 64k pagesize stretches the limits even further.   Here a two level
> > pagetable tree would cover 4TB, 3-level could cover 32PB exceeding
> > the capacity of every MIPS processor ever made - and probably sufficient
> > for the coming decade :-)
> 
>  Further increasing of the page size should result in better performance
> due to fewer TLB misses and reduce the memory footprint of page tables,
> but the drawback is more memory is wasted for maps.  Whether the end
> result is a gain or a loss depends on the actual application of a system,
> so I guess we should either leave the size configurable (with a sane
> default for those who might have troubles judging what would suit them
> best) or only decide on a given size after lots of benchmarking.

Unless somebody yells I almost feel like ditching 3-level pagetable
support; 2-levels with a decent pagesize should suffice for a few years
to come ...

  Ralf

Re: MIPS addressing limits, Was: Re: CVS Update@-mips.org: linux

[Jun Sun]

On Wed, Oct 15, 2003 at 04:59:48PM +0200, Ralf Baechle wrote:
> On Wed, Oct 15, 2003 at 04:23:06PM +0200, Maciej W. Rozycki wrote:
> 
> > > Still want more?  A 3 level tree would then cover 128TB of virtual
> > > address space already exceedin the hardware limits of all processors but
> > > the R8000.
> > 
> >  Well, the MIPS64 ISA spec allows up to 8EB of user memory to be supported
> > by an implementation, IIRC; probably nothing supports that much yet,
> > though. ;-)  BTW, is an R8000 spec available online anywhere?
> 
> There used to be a few papers published by SGI online and various other
> bits of information I found through google.
> 
> (I happen to have a paper copy of the R8000 manual but since the responsible
> people still haven't informed me if I can legally use it, this book is
> closed and will stay closed until this happens - if ever ...  Pitty, I
> still receive mails from various R8000 users ...)
> 
> > > 64k pagesize stretches the limits even further.   Here a two level
> > > pagetable tree would cover 4TB, 3-level could cover 32PB exceeding
> > > the capacity of every MIPS processor ever made - and probably sufficient
> > > for the coming decade :-)
> > 
> >  Further increasing of the page size should result in better performance
> > due to fewer TLB misses and reduce the memory footprint of page tables,
> > but the drawback is more memory is wasted for maps.  Whether the end
> > result is a gain or a loss depends on the actual application of a system,
> > so I guess we should either leave the size configurable (with a sane
> > default for those who might have troubles judging what would suit them
> > best) or only decide on a given size after lots of benchmarking.
> 
> Unless somebody yells I almost feel like ditching 3-level pagetable
> support; 2-levels with a decent pagesize should suffice for a few years
> to come ...
> 

Isn't ia64 still using 3-level page tables?  Any performance data we
can infer from theirs?

I feel a little uneasy about ditching 3-level pagetable altogether.
Leaving all the parameters configurable, including the possiblity of
nullifying the second level and changing page size, seems to be a more 
comforting thought.

Jun

Re: MIPS addressing limits, Was: Re: CVS Update@-mips.org: linux

[Ralf Baechle]

On Wed, Oct 15, 2003 at 10:19:02AM -0700, Jun Sun wrote:

> Isn't ia64 still using 3-level page tables?  Any performance data we
> can infer from theirs?

Very different MMU, IA64 data is hardly an indicator.  And anyway, the
result should be MIPS TLB refill handlers suck big chunky rocks through
a straw so should be rewritten ;-)

> I feel a little uneasy about ditching 3-level pagetable altogether.
> Leaving all the parameters configurable, including the possiblity of
> nullifying the second level and changing page size, seems to be a more 
> comforting thought.

3 levels are only needed if you can seriously say you're need more than
64GB of vmalloc space or processes larger than 64GB.  Little need for
that in the current universe though I know one institution which broke
the 1TB process size limit > 5 years ago.

  Ralf

Re: MIPS addressing limits, Was: Re: CVS Update@-mips.org: linux

[Nick]

If anyone wants to poke said folks into handing in the OK I'll ship
someone an R8k I2 to work on.
	Nick

On Wed, 15 Oct 2003, Ralf Baechle wrote:

> On Wed, Oct 15, 2003 at 04:23:06PM +0200, Maciej W. Rozycki wrote:
> 
> > > Still want more?  A 3 level tree would then cover 128TB of virtual
> > > address space already exceedin the hardware limits of all processors but
> > > the R8000.
> > 
> >  Well, the MIPS64 ISA spec allows up to 8EB of user memory to be supported
> > by an implementation, IIRC; probably nothing supports that much yet,
> > though. ;-)  BTW, is an R8000 spec available online anywhere?
> 
> There used to be a few papers published by SGI online and various other
> bits of information I found through google.
> 
> (I happen to have a paper copy of the R8000 manual but since the responsible
> people still haven't informed me if I can legally use it, this book is
> closed and will stay closed until this happens - if ever ...  Pitty, I
> still receive mails from various R8000 users ...)
> 
> > > 64k pagesize stretches the limits even further.   Here a two level
> > > pagetable tree would cover 4TB, 3-level could cover 32PB exceeding
> > > the capacity of every MIPS processor ever made - and probably sufficient
> > > for the coming decade :-)
> > 
> >  Further increasing of the page size should result in better performance
> > due to fewer TLB misses and reduce the memory footprint of page tables,
> > but the drawback is more memory is wasted for maps.  Whether the end
> > result is a gain or a loss depends on the actual application of a system,
> > so I guess we should either leave the size configurable (with a sane
> > default for those who might have troubles judging what would suit them
> > best) or only decide on a given size after lots of benchmarking.
> 
> Unless somebody yells I almost feel like ditching 3-level pagetable
> support; 2-levels with a decent pagesize should suffice for a few years
> to come ...
> 
>   Ralf
>