Does Linux invalidate TLB entries?

Does Linux invalidate TLB entries?

[Matthew Dharm]

I'm still trying to track down the cause of all my problems, so I'm
going over the RM7000 errata.

I see one here that I'm not sure if it's a problem or not.  It only
applies to OSes which invalidate TLB entries and thus will cause TLB
Invalid exceptions (as opposed to a TLB refill exception, I think).

So, does Linux invalidate TLBs?  I've been looking at the code, and I
think the answer is 'no', but I'm not really sure.

Matt

--
Matthew D. Dharm                            Senior Software Designer
Momentum Computer Inc.                      1815 Aston Ave.  Suite 107
(760) 431-8663 X-115                        Carlsbad, CA 92008-7310
Momentum Works For You                      www.momenco.com

Re: Does Linux invalidate TLB entries?

[Ralf Baechle]

On Tue, Jan 29, 2002 at 05:03:42PM -0800, Matthew Dharm wrote:

> I'm still trying to track down the cause of all my problems, so I'm
> going over the RM7000 errata.
> 
> I see one here that I'm not sure if it's a problem or not.  It only
> applies to OSes which invalidate TLB entries and thus will cause TLB
> Invalid exceptions (as opposed to a TLB refill exception, I think).
> 
> So, does Linux invalidate TLBs?  I've been looking at the code, and I
> think the answer is 'no', but I'm not really sure.

Yes, Linux may create TLB entries with V=0.

  Ralf

RE: Does Linux invalidate TLB entries?

[Matthew Dharm]

Damn.  The entire line of processors from the RM7000 to the 7000A,
7000B, 7061A, and 7065A all have a bug which involves invalid TLB
entries.

I've sent the errata to Ralf only for review.  Basically, under
certain circumstances the processor will take the "TLB refill"
exception vector instead of the "TLB invalid" vector.

Matt

--
Matthew D. Dharm                            Senior Software Designer
Momentum Computer Inc.                      1815 Aston Ave.  Suite 107
(760) 431-8663 X-115                        Carlsbad, CA 92008-7310
Momentum Works For You                      www.momenco.com

> -----Original Message-----
> From: Ralf Baechle [mailto:ralf@oss.sgi.com]
> Sent: Wednesday, January 30, 2002 3:03 AM
> To: Matthew Dharm
> Cc: Linux-MIPS
> Subject: Re: Does Linux invalidate TLB entries?
>
>
> On Tue, Jan 29, 2002 at 05:03:42PM -0800, Matthew Dharm wrote:
>
> > I'm still trying to track down the cause of all my
> problems, so I'm
> > going over the RM7000 errata.
> >
> > I see one here that I'm not sure if it's a problem or
> not.  It only
> > applies to OSes which invalidate TLB entries and thus
> will cause TLB
> > Invalid exceptions (as opposed to a TLB refill exception,
> I think).
> >
> > So, does Linux invalidate TLBs?  I've been looking at the
> code, and I
> > think the answer is 'no', but I'm not really sure.
>
> Yes, Linux may create TLB entries with V=0.
>
>   Ralf
>

RE: Does Linux invalidate TLB entries?

[Justin Carlson]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

On Wed, 2002-01-30 at 14:33, Matthew Dharm wrote:
> Damn.  The entire line of processors from the RM7000 to the 7000A,
> 7000B, 7061A, and 7065A all have a bug which involves invalid TLB
> entries.
> 
> I've sent the errata to Ralf only for review.  Basically, under
> certain circumstances the processor will take the "TLB refill"
> exception vector instead of the "TLB invalid" vector.

What's the behavior if the invalid entry is not fixed up and we replay
the offending instruction?  If there's a guarantee that it won't take
the wrong vector repeatedly, then this would be trivial to fix (and may
not need one at all for correctness).

-Justin


[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

RE: Does Linux invalidate TLB entries?

[Matthew Dharm]

The errata, unfortunately, doen't say.

It does say that the suggested workaround is to use the TLBP operation
to look for a matching but invalid entry, and then branch to the
invalid handler if necessary.

It also says that the CP0 Cause, EPC, BadVaddr and ENHI will wold the
values for the dstream TLB exception.  In other words, it's all set up
for the invalid exception, but it jumps to the refill exception
instead.

Matt

--
Matthew D. Dharm                            Senior Software Designer
Momentum Computer Inc.                      1815 Aston Ave.  Suite 107
(760) 431-8663 X-115                        Carlsbad, CA 92008-7310
Momentum Works For You                      www.momenco.com

> -----Original Message-----
> From: Justin Carlson [mailto:justincarlson@cmu.edu]
> Sent: Wednesday, January 30, 2002 12:45 PM
> To: Matthew Dharm
> Cc: linux-mips@oss.sgi.com
> Subject: RE: Does Linux invalidate TLB entries?
>
>
> On Wed, 2002-01-30 at 14:33, Matthew Dharm wrote:
> > Damn.  The entire line of processors from the RM7000 to the 7000A,
> > 7000B, 7061A, and 7065A all have a bug which involves invalid TLB
> > entries.
> >
> > I've sent the errata to Ralf only for review.  Basically, under
> > certain circumstances the processor will take the "TLB refill"
> > exception vector instead of the "TLB invalid" vector.
>
> What's the behavior if the invalid entry is not fixed up
> and we replay
> the offending instruction?  If there's a guarantee that it
> won't take
> the wrong vector repeatedly, then this would be trivial to
> fix (and may
> not need one at all for correctness).
>
> -Justin
>
>

RE: Does Linux invalidate TLB entries?

[Matthew Dharm]

The errata, unfortunately, doen't say.

It does say that the suggested workaround is to use the TLBP operation
to look for a matching but invalid entry, and then branch to the
invalid handler if necessary.

It also says that the CP0 Cause, EPC, BadVaddr and ENHI will wold the
values for the dstream TLB exception.  In other words, it's all set up
for the invalid exception, but it jumps to the refill exception
instead.

Matt

--
Matthew D. Dharm                            Senior Software Designer
Momentum Computer Inc.                      1815 Aston Ave.  Suite 107
(760) 431-8663 X-115                        Carlsbad, CA 92008-7310
Momentum Works For You                      www.momenco.com

> -----Original Message-----
> From: Justin Carlson [mailto:justincarlson@cmu.edu]
> Sent: Wednesday, January 30, 2002 12:45 PM
> To: Matthew Dharm
> Cc: linux-mips@oss.sgi.com
> Subject: RE: Does Linux invalidate TLB entries?
>
>
> On Wed, 2002-01-30 at 14:33, Matthew Dharm wrote:
> > Damn.  The entire line of processors from the RM7000 to the 7000A,
> > 7000B, 7061A, and 7065A all have a bug which involves invalid TLB
> > entries.
> >
> > I've sent the errata to Ralf only for review.  Basically, under
> > certain circumstances the processor will take the "TLB refill"
> > exception vector instead of the "TLB invalid" vector.
>
> What's the behavior if the invalid entry is not fixed up
> and we replay
> the offending instruction?  If there's a guarantee that it
> won't take
> the wrong vector repeatedly, then this would be trivial to
> fix (and may
> not need one at all for correctness).
>
> -Justin
>
>

RE: Does Linux invalidate TLB entries?

[Justin Carlson]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

On Wed, 2002-01-30 at 16:23, Matthew Dharm wrote:
> The errata, unfortunately, doen't say.
> 
> It does say that the suggested workaround is to use the TLBP operation
> to look for a matching but invalid entry, and then branch to the
> invalid handler if necessary.
> 
> It also says that the CP0 Cause, EPC, BadVaddr and ENHI will wold the
> values for the dstream TLB exception.  In other words, it's all set up
> for the invalid exception, but it jumps to the refill exception
> instead.

Looking at the linux side a bit closer, I don't think this can ever hit
us.  All the MIPS variants (that I'm familiar with) only use invalid TLB
entries to flush the TLB, and in that case they set up the entries to be
in KSEG0.  

The only way to trigger an invalid TLB op, then, is to try to do a load
from a proper range in KSEG0 after a flush; in that case the right thing
for the processor to do not a TLB invalid exception anyways, as there is
no TLB-based translation required.  We'd still segfault and die as a
user process, of course, since you can't touch KSEG0, but that's good
and proper.

Anyone else see another case that would be a problem for us?

-Justin


[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Does Linux invalidate TLB entries?

[Dominic Sweetman]

Matthew Dharm (mdharm@momenco.com) writes:

> I see one here that I'm not sure if it's a problem or not.  It only
> applies to OSes which invalidate TLB entries and thus will cause TLB
> Invalid exceptions (as opposed to a TLB refill exception, I think).

Hmm.  I see that the triggering condition is related to the
instruction stream (you need an ITLB miss and an icache miss).

> So, does Linux invalidate TLBs?  I've been looking at the code, and I
> think the answer is 'no', but I'm not really sure.

RM7000 (like most MIPS CPUs) maps pages in pairs: one virtual address
tag (covering 8Kbyte of virtual address space) has mappings to two
4K physical pages.

Sure, quite a lot of the time both pages will be mapped: but unless
the Linux VM system does everything in pairs of pages (unlikely) it
will sometimes happen that you only have a mapping for one 4K page:
the other mapping will be invalid, and then this problem might happen.

When it does, the TLB refill handler will blindly load a second copy
of the same translation (still half-invalid, I guess).

PMC's statement that "Most operating systems do not invalidate TLB
entries, so, in those systems, this errata will not be seen..." seems
like wishful thinking.  Unless "most operating systems" don't do
virtual memory at all, of course.

If my argument is sound it leaves only the TLBP solution, which is
not very nice.

-- 
Dominic Sweetman, 
Algorithmics Ltd
The Fruit Farm, Ely Road, Chittering, CAMBS CB5 9PH, ENGLAND
phone: +44 1223 706200 / fax: +44 1223 706250 / direct: +44 1223 706205
http://www.algor.co.uk