Large TLBs on 40x

Large TLBs on 40x

[Josh Boyer]

Howdy,

In 2.6, the PPC mm code for 4xx platforms uses large TLB entries for
most of RAM.  Specifically, I am looking at arch/ppc/mm/4xx_mmu.c at the
mmu_mapin_ram function.  In there, the pmd pointers are setup with
_PMD_SIZE_16M | _PAGE_HWEXEC | _PAGE_HWWRITE.

Doesn't this allow the text pages of the kernel to be written to?  So to
my understanding, a buggy driver that went off into the weeds could
corrupt the text pages of the kernel.

I would think that you would want the kernel text pages to be
non-writable so that if something like the above scenario happened you
don't have possibly bad text pages.  Debugging wouldn't be much fun.

I think I am missing something here.  If so, please excuse my lack of
knowledge on TLBs, but I am somewhat new to the low level stuff.  Anyone
care to point out what I am missing?

thx,
josh


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Matt Porter]

On Tue, Aug 03, 2004 at 06:05:42PM -0500, Josh Boyer wrote:
>
> Howdy,
>
> In 2.6, the PPC mm code for 4xx platforms uses large TLB entries for
> most of RAM.  Specifically, I am looking at arch/ppc/mm/4xx_mmu.c at the
> mmu_mapin_ram function.  In there, the pmd pointers are setup with
> _PMD_SIZE_16M | _PAGE_HWEXEC | _PAGE_HWWRITE.
>
> Doesn't this allow the text pages of the kernel to be written to?  So to

Yes.

> my understanding, a buggy driver that went off into the weeds could
> corrupt the text pages of the kernel.

Yes.

> I would think that you would want the kernel text pages to be
> non-writable so that if something like the above scenario happened you
> don't have possibly bad text pages.  Debugging wouldn't be much fun.

Ideally, that's true. However, the kernel lowmem large TLB mapping is
there for performance. There's no way to manage default page size
protection on the text pages. It's a tradeoff between performance
and the ease of debugging a misguided driver. With all the kernel
debugging tools these days, it sure seems like performance is the
clear winner.

> I think I am missing something here.  If so, please excuse my lack of
> knowledge on TLBs, but I am somewhat new to the low level stuff.  Anyone
> care to point out what I am missing?

Well, it should be noted that the Classic PPC mapin_ram does a similar
thing. When mapping with BATs, the ability to do per (default size) page
RO is not available.  For reasons of performance, this is the default
method to map kernel lowmem on Classic PPC cores.

The same situation is present on Book E parts (44x/85xx). Since they
map all of kernel lowmem staticly with large TLB entries, they also
lack the ability to cover kernel text with read-only entries.

Anyway, this isn't just 4xx, it's pretty much everywhere. It's
possible to make 4xx large TLB mapping optional, but it's not
clear that adding the option is really that useful (at least
to me, YMMV).

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Josh Boyer]

On Tue, 2004-08-03 at 19:16, Matt Porter wrote:
> > I would think that you would want the kernel text pages to be
> > non-writable so that if something like the above scenario happened you
> > don't have possibly bad text pages.  Debugging wouldn't be much fun.
>
> Ideally, that's true. However, the kernel lowmem large TLB mapping is
> there for performance. There's no way to manage default page size
> protection on the text pages. It's a tradeoff between performance
> and the ease of debugging a misguided driver. With all the kernel
> debugging tools these days, it sure seems like performance is the
> clear winner.

Well at least my understanding of what was going on was right :).  I see
the tradeoff, I was just wondering if there was some happy medium.
Maybe pinning a read-only TLB entry for most of the text pages for
however large you can get would work...  or maybe not.

It's probably not that large of an issue for a normal user because the
kernel and drivers are fairly stable, which makes performance more
important.  But when someone is writing new drivers, randomly poking
holes in the kernel text pages and then executing instructions from
there leads to some strange panics.  Although, the new driver should be
suspect in that case anyway :).

>
> > I think I am missing something here.  If so, please excuse my lack of
> > knowledge on TLBs, but I am somewhat new to the low level stuff.  Anyone
> > care to point out what I am missing?
>
> Well, it should be noted that the Classic PPC mapin_ram does a similar
> thing. When mapping with BATs, the ability to do per (default size) page
> RO is not available.  For reasons of performance, this is the default
> method to map kernel lowmem on Classic PPC cores.
>
> The same situation is present on Book E parts (44x/85xx). Since they
> map all of kernel lowmem staticly with large TLB entries, they also
> lack the ability to cover kernel text with read-only entries.
>
> Anyway, this isn't just 4xx, it's pretty much everywhere. It's
> possible to make 4xx large TLB mapping optional, but it's not
> clear that adding the option is really that useful (at least
> to me, YMMV).

Yeah, I figured it was more than just 40x, it's just that is what I am
using right now.  Thanks for the info.

josh


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 4, 2004, at 7:49 AM, Josh Boyer wrote:

> Maybe pinning a read-only TLB entry for most of the text pages for
> however large you can get would work...  or maybe not.

The problem is BATs and pinned TLBs require alignment restrictions
that don't allow this.

> .....But when someone is writing new drivers, randomly poking
> holes in the kernel text pages and then executing instructions from
> there leads to some strange panics.  Although, the new driver should be
> suspect in that case anyway :).

Then, on MPC8xx don't select pinned TLBs to cover the kernel
space, and on traditional PPC MMUs select  the 'nobat' option
on the command line.  This will write protect the kernel text pages.
However, if you select something like CONFIG_KGDB, this will
allow writing of the text pages to set breakpoints.  You will need
to use a debugger like the BDI2000 and use hardware breakpoints
to do your debugging.  When you are done, you can enable the
performance TLB options again.  I thought the 4xx allowed
something like this as well.  If not, it could and probably should.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Matt Porter]

On Wed, Aug 04, 2004 at 12:36:56PM -0400, Dan Malek wrote:
>
> On Aug 4, 2004, at 7:49 AM, Josh Boyer wrote:
>
> > Maybe pinning a read-only TLB entry for most of the text pages for
> > however large you can get would work...  or maybe not.
>
> The problem is BATs and pinned TLBs require alignment restrictions
> that don't allow this.
>
> > .....But when someone is writing new drivers, randomly poking
> > holes in the kernel text pages and then executing instructions from
> > there leads to some strange panics.  Although, the new driver should be
> > suspect in that case anyway :).
>
> Then, on MPC8xx don't select pinned TLBs to cover the kernel
> space, and on traditional PPC MMUs select  the 'nobat' option
> on the command line.  This will write protect the kernel text pages.
> However, if you select something like CONFIG_KGDB, this will
> allow writing of the text pages to set breakpoints.  You will need
> to use a debugger like the BDI2000 and use hardware breakpoints
> to do your debugging.  When you are done, you can enable the
> performance TLB options again.  I thought the 4xx allowed
> something like this as well.  If not, it could and probably should.

We eagerly await your patch. ;)

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 4, 2004, at 12:39 PM, Matt Porter wrote:

> We eagerly await your patch. ;)


HAHAHAHA!   Fortunately, long ago IBM asked me to return
all of the development boards I had that belonged to them........
and I had some cool ones, too :-)

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Josh Boyer]

On Wed, 2004-08-04 at 12:03, Dan Malek wrote:
> On Aug 4, 2004, at 12:39 PM, Matt Porter wrote:
>
> > We eagerly await your patch. ;)
>
>
> HAHAHAHA!   Fortunately, long ago IBM asked me to return
> all of the development boards I had that belonged to them........
> and I had some cool ones, too :-)

(Sorry for the delay)...

I have some boards I could test such a patch on.  Or is it a case of
"this isn't trivial to do"?  In either case, if someone could give me
some pointers, I'll try and figure it out myself.

And if not, I guess I'll still figure it out myself, but it'll take a
lot longer :).

thx,
josh


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 9, 2004, at 10:21 PM, Josh Boyer wrote:

> I have some boards I could test such a patch on.  Or is it a case of
> "this isn't trivial to do"?

Ummmmm.....what are we talking about here?

Both MPC8xx, and IBM40x have configuration options for
pinning some (small) amount of kernel space.  You can choose
to enable this if you wish, I don't think it is normally enabled.
I've never found a benchmark that proved either was better,
but I left the code there for others to experiment with.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Matt Porter]

On Tue, Aug 10, 2004 at 02:04:44AM -0400, Dan Malek wrote:
>
> On Aug 9, 2004, at 10:21 PM, Josh Boyer wrote:
>
> > I have some boards I could test such a patch on.  Or is it a case of
> > "this isn't trivial to do"?
>
> Ummmmm.....what are we talking about here?
>
> Both MPC8xx, and IBM40x have configuration options for
> pinning some (small) amount of kernel space.  You can choose
> to enable this if you wish, I don't think it is normally enabled.
> I've never found a benchmark that proved either was better,
> but I left the code there for others to experiment with.

PPC40x no longer has this in 2.6. It has been deprecated by the dynamic
large tlb support. He's looking for an option (like nobats) to cause
kernel lowmem to not be mapped by large page entries.

So, to answer the original question, I would suggest a patch
that that uses a 'noltlb' cmdline option, then skips the PPC40x
large page mapping in 4xx_mmu.c based on that. See the code
that implements 'nobats' for an example.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 10, 2004, at 10:35 AM, Matt Porter wrote:

> PPC40x no longer has this in 2.6.

Oh, I see now.

> So, to answer the original question, I would suggest a patch
> that that uses a 'noltlb' cmdline option,

Yes, that would work.

Thanks for the info.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Josh Boyer]

On Tue, 2004-08-10 at 09:35, Matt Porter wrote:
> On Tue, Aug 10, 2004 at 02:04:44AM -0400, Dan Malek wrote:
> >
> > On Aug 9, 2004, at 10:21 PM, Josh Boyer wrote:
> >
> > > I have some boards I could test such a patch on.  Or is it a case of
> > > "this isn't trivial to do"?
> >
> > Ummmmm.....what are we talking about here?
> >
> > Both MPC8xx, and IBM40x have configuration options for
> > pinning some (small) amount of kernel space.  You can choose
> > to enable this if you wish, I don't think it is normally enabled.
> > I've never found a benchmark that proved either was better,
> > but I left the code there for others to experiment with.
>
> PPC40x no longer has this in 2.6. It has been deprecated by the dynamic
> large tlb support. He's looking for an option (like nobats) to cause
> kernel lowmem to not be mapped by large page entries.
>
> So, to answer the original question, I would suggest a patch
> that that uses a 'noltlb' cmdline option, then skips the PPC40x
> large page mapping in 4xx_mmu.c based on that. See the code
> that implements 'nobats' for an example.

Hm...  like this?  Against 2.6.8-rc4.

thx,
josh


diff -Naur -x '*.swp' linux-2.6.orig/arch/ppc/mm/4xx_mmu.c linux-2.6/arch/ppc/mm/4xx_mmu.c
--- linux-2.6.orig/arch/ppc/mm/4xx_mmu.c	2004-06-16 00:18:37.000000000 -0500
+++ linux-2.6/arch/ppc/mm/4xx_mmu.c	2004-08-11 08:25:11.000000000 -0500
@@ -52,6 +52,7 @@
 #include <asm/setup.h>
 #include "mmu_decl.h"

+extern int __map_without_ltlbs;
 /*
  * MMU_init_hw does the chip-specific initialization of the MMU hardware.
  */
@@ -102,6 +103,10 @@
 	p = PPC_MEMSTART;
 	s = 0;

+	if (__map_without_ltlbs) {
+		return s;
+	}
+
 	while (s <= (total_lowmem - LARGE_PAGE_SIZE_16M)) {
 		pmd_t *pmdp;
 		unsigned long val = p | _PMD_SIZE_16M | _PAGE_HWEXEC | _PAGE_HWWRITE;
diff -Naur -x '*.swp' linux-2.6.orig/arch/ppc/mm/init.c linux-2.6/arch/ppc/mm/init.c
--- linux-2.6.orig/arch/ppc/mm/init.c	2004-08-11 08:03:55.000000000 -0500
+++ linux-2.6/arch/ppc/mm/init.c	2004-08-11 08:11:24.000000000 -0500
@@ -104,6 +104,7 @@
  * -- Cort
  */
 int __map_without_bats;
+int __map_without_ltlbs;

 /* max amount of RAM to use */
 unsigned long __max_memory;
@@ -204,6 +205,10 @@
 		__map_without_bats = 1;
 	}

+	if (strstr(cmd_line, "noltlb")) {
+		__map_without_ltlbs = 1;
+	}
+
 	/* Look for mem= option on command line */
 	if (strstr(cmd_line, "mem=")) {
 		char *p, *q;


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 11, 2004, at 9:50 AM, Josh Boyer wrote:

> Hm...  like this?  Against 2.6.8-rc4.

Does it work the way you want it now?

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Josh Boyer]

On Wed, 2004-08-11 at 10:30, Dan Malek wrote:
> On Aug 11, 2004, at 9:50 AM, Josh Boyer wrote:
>
> > Hm...  like this?  Against 2.6.8-rc4.
>
> Does it work the way you want it now?

Yep, tested it on my board.  It's not the end goal I am looking for, but
it's a good starting point.

Basically, what I would like to get eventually is large TLBs + protected
kernel text pages.  Like you and Matt pointed out, there are issues with
that.  If it can't be done, then at least I can use this patch to
protect the kernel text pages easily.  Sometimes protection is more
important than performance.  Especially when I am hacking on stuff ;).

Thanks for the help.

josh


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 11, 2004, at 1:58 PM, Josh Boyer wrote:

> Yep, tested it on my board.  It's not the end goal I am looking for,
> but
> it's a good starting point.

Ummm...OK.

> Basically, what I would like to get eventually is large TLBs +
> protected
> kernel text pages.  Like you and Matt pointed out, there are issues
> with
> that.  If it can't be done, .....

It can be done, just not easily.  Requires boundary/alignment of
text/data
and set up of the large TLBs.  Then, there is the issue with wanting the
same feature with loadable modules......

> .......  Sometimes protection is more
> important than performance.  Especially when I am hacking on stuff ;).

Well, you have the option now.  Have fun!


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Matt Porter]

On Wed, Aug 11, 2004 at 08:50:03AM -0500, Josh Boyer wrote:
> On Tue, 2004-08-10 at 09:35, Matt Porter wrote:
> > On Tue, Aug 10, 2004 at 02:04:44AM -0400, Dan Malek wrote:
> > >
> > > On Aug 9, 2004, at 10:21 PM, Josh Boyer wrote:
> > >
> > > > I have some boards I could test such a patch on.  Or is it a case of
> > > > "this isn't trivial to do"?
> > >
> > > Ummmmm.....what are we talking about here?
> > >
> > > Both MPC8xx, and IBM40x have configuration options for
> > > pinning some (small) amount of kernel space.  You can choose
> > > to enable this if you wish, I don't think it is normally enabled.
> > > I've never found a benchmark that proved either was better,
> > > but I left the code there for others to experiment with.
> >
> > PPC40x no longer has this in 2.6. It has been deprecated by the dynamic
> > large tlb support. He's looking for an option (like nobats) to cause
> > kernel lowmem to not be mapped by large page entries.
> >
> > So, to answer the original question, I would suggest a patch
> > that that uses a 'noltlb' cmdline option, then skips the PPC40x
> > large page mapping in 4xx_mmu.c based on that. See the code
> > that implements 'nobats' for an example.
>
> Hm...  like this?  Against 2.6.8-rc4.

Looks good to me, except we should probably have the cmdline option
be 'noltlbs'...minor nit.

Please repost after reading http://kerneltrap.org/node/view/3180
and agreeing to the DCO by providing a signed off line. Then it
can go into mainline.

Thanks,
Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Josh Boyer]

On Wed, 2004-08-11 at 10:45, Matt Porter wrote:
> On Wed, Aug 11, 2004 at 08:50:03AM -0500, Josh Boyer wrote:
> > On Tue, 2004-08-10 at 09:35, Matt Porter wrote:
> > > On Tue, Aug 10, 2004 at 02:04:44AM -0400, Dan Malek wrote:
> > > >
> > > > On Aug 9, 2004, at 10:21 PM, Josh Boyer wrote:
> > > >
> > > > > I have some boards I could test such a patch on.  Or is it a case of
> > > > > "this isn't trivial to do"?
> > > >
> > > > Ummmmm.....what are we talking about here?
> > > >
> > > > Both MPC8xx, and IBM40x have configuration options for
> > > > pinning some (small) amount of kernel space.  You can choose
> > > > to enable this if you wish, I don't think it is normally enabled.
> > > > I've never found a benchmark that proved either was better,
> > > > but I left the code there for others to experiment with.
> > >
> > > PPC40x no longer has this in 2.6. It has been deprecated by the dynamic
> > > large tlb support. He's looking for an option (like nobats) to cause
> > > kernel lowmem to not be mapped by large page entries.
> > >
> > > So, to answer the original question, I would suggest a patch
> > > that that uses a 'noltlb' cmdline option, then skips the PPC40x
> > > large page mapping in 4xx_mmu.c based on that. See the code
> > > that implements 'nobats' for an example.
> >
> > Hm...  like this?  Against 2.6.8-rc4.
>
> Looks good to me, except we should probably have the cmdline option
> be 'noltlbs'...minor nit.

Yep, fixed.  Thanks for the tips.

thx,
josh

Signed-off-by: Josh Boyer

diff -Naur -x '*.swp' linux-2.6.orig/arch/ppc/mm/4xx_mmu.c linux-2.6/arch/ppc/mm/4xx_mmu.c
--- linux-2.6.orig/arch/ppc/mm/4xx_mmu.c	2004-06-16 00:18:37.000000000 -0500
+++ linux-2.6/arch/ppc/mm/4xx_mmu.c	2004-08-11 08:25:11.000000000 -0500
@@ -52,6 +52,7 @@
 #include <asm/setup.h>
 #include "mmu_decl.h"

+extern int __map_without_ltlbs;
 /*
  * MMU_init_hw does the chip-specific initialization of the MMU hardware.
  */
@@ -102,6 +103,10 @@
 	p = PPC_MEMSTART;
 	s = 0;

+	if (__map_without_ltlbs) {
+		return s;
+	}
+
 	while (s <= (total_lowmem - LARGE_PAGE_SIZE_16M)) {
 		pmd_t *pmdp;
 		unsigned long val = p | _PMD_SIZE_16M | _PAGE_HWEXEC | _PAGE_HWWRITE;
diff -Naur -x '*.swp' linux-2.6.orig/arch/ppc/mm/init.c linux-2.6/arch/ppc/mm/init.c
--- linux-2.6.orig/arch/ppc/mm/init.c	2004-08-11 08:03:55.000000000 -0500
+++ linux-2.6/arch/ppc/mm/init.c	2004-08-11 08:11:24.000000000 -0500
@@ -104,6 +104,7 @@
  * -- Cort
  */
 int __map_without_bats;
+int __map_without_ltlbs;

 /* max amount of RAM to use */
 unsigned long __max_memory;
@@ -204,6 +205,10 @@
 		__map_without_bats = 1;
 	}

+	if (strstr(cmd_line, "noltlbs")) {
+		__map_without_ltlbs = 1;
+	}
+
 	/* Look for mem= option on command line */
 	if (strstr(cmd_line, "mem=")) {
 		char *p, *q;


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Dan Malek]

On Aug 11, 2004, at 1:42 PM, Josh Boyer wrote:

> Yep, fixed.  Thanks for the tips.

Although a single sentence, my last question was meant
to be serious.  The changes here look reasonable, but
does the behavior change to be what you want to see?
What happens if you configure with/without KGDB
enabled (since this affects protection attributes)?

As a maintainer, I just can't accept a code patch without
documentation of the problem and that it is corrected.
I don't have the time to test all of these conditions before
pushing patches to the source tree, that's _your_ job :-)

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Large TLBs on 40x

[Matt Porter]

On Wed, Aug 11, 2004 at 02:16:20PM -0400, Dan Malek wrote:
>
> On Aug 11, 2004, at 1:42 PM, Josh Boyer wrote:
>
> > Yep, fixed.  Thanks for the tips.
>
> Although a single sentence, my last question was meant
> to be serious.  The changes here look reasonable, but
> does the behavior change to be what you want to see?
> What happens if you configure with/without KGDB
> enabled (since this affects protection attributes)?

Well, we know what happens there.  The protection goes away since
we have code to ensure that kernel text is writeable when we have
any of the debuggers enabled...regardless of whether notlbs or
nobats is set.

> As a maintainer, I just can't accept a code patch without
> documentation of the problem and that it is corrected.
> I don't have the time to test all of these conditions before
> pushing patches to the source tree, that's _your_ job :-)

In any case, I've heard from folks that don't want large tlb
mappings in some cases, so this little unobtrustive option
has other uses.

-Matt

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/