TLB miss handler code

TLB miss handler code

[Nawab Ali]

Hi,
I have a Sun Ultra-10 running Debian woody. I need to modify the TLB miss
handler code. Basically everytime there is a TLB miss, I'm trying to log
the new PTE that is brought into the TLB.

I know that SPARC machines handle TLB refills in software. Can someone
please point me to the TLB miss handler code(which file???) in the linux
kernel(2.4.24)??? I'm a bit lost here.

Any help would be greatly appreciated.

Thanx.
nawab

Re: TLB miss handler code

[Ed L Cashin]

Nawab Ali <alin@ececs.uc.edu> writes:

> Hi,
> I have a Sun Ultra-10 running Debian woody. I need to modify the TLB miss
> handler code. Basically everytime there is a TLB miss, I'm trying to log
> the new PTE that is brought into the TLB.
>
> I know that SPARC machines handle TLB refills in software. Can someone
> please point me to the TLB miss handler code(which file???) in the linux
> kernel(2.4.24)??? I'm a bit lost here.
>
> Any help would be greatly appreciated.

Well, I am not sure about 2.4.24, but in 2.6.0, the do_sparc64_fault
routine in arch/sparc64/mm/fault.c is invoked during TLB miss
handling.  When it's a TLB miss you can tell by the fact that
(fault_code & FAULT_CODE_DTLB || fault_code & FAULT_CODE_ITLB).

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[Ben Collins]

> My question is, where has __builtin_trap() been implemented?? Also is
> __builtin_trap() where the missing PTE is picked up from the page table and put in the
> TLB because that's exactly what I'm looking for??

All the __builtin_* functions are defined in gcc. This one happens to
turn into "ta 5". Basically forces an oops in the kernel. So no, it does
not cause anything like what you are looking for.

-- 
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
WatchGuard - http://www.watchguard.com/

Re: TLB miss handler code

[Nawab Ali]

Hi Ed,
Thanx for pointing out the file. I checked out the do_sparc64_fault
function in arch/sparc64/mm/fault.c (2.6.0).

This is the code snippet I found.
if ((fault_code & FAULT_CODE_ITLB) && (fault_code & FAULT_CODE_DTLB))
	BUG();

-----
BUG() has been defined as following in
linux-2.6.0/include/asm-sparc64/bug.h

#ifdef CONFIG_DEBUG_BUGVERBOSE
extern void do_BUG(const char *file, int line);
#define BUG() do {                                      \
        do_BUG(__FILE__, __LINE__);                     \
        __builtin_trap();                               \
} while (0)
#else
#define BUG()           __builtin_trap()
#endif
------

My question is, where has __builtin_trap() been implemented?? Also is
__builtin_trap() where the missing PTE is picked up from the page table and put in the
TLB because that's exactly what I'm looking for??

Thanx
nawab


On Mon, 19 Jan 2004, Ed L Cashin wrote:

> Nawab Ali <alin@ececs.uc.edu> writes:
>
> > Hi,
> > I have a Sun Ultra-10 running Debian woody. I need to modify the TLB miss
> > handler code. Basically everytime there is a TLB miss, I'm trying to log
> > the new PTE that is brought into the TLB.
> >
> > I know that SPARC machines handle TLB refills in software. Can someone
> > please point me to the TLB miss handler code(which file???) in the linux
> > kernel(2.4.24)??? I'm a bit lost here.
> >
> > Any help would be greatly appreciated.
>
> Well, I am not sure about 2.4.24, but in 2.6.0, the do_sparc64_fault
> routine in arch/sparc64/mm/fault.c is invoked during TLB miss
> handling.  When it's a TLB miss you can tell by the fact that
> (fault_code & FAULT_CODE_DTLB || fault_code & FAULT_CODE_ITLB).
>
> --
> --Ed L Cashin            |   PGP public key:
>   ecashin@uga.edu        |   http://noserose.net/e/pgp/
>
> -
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

==
Ask not what Mozilla can do for you...
---------------------------------------------------------
  Nawab Ali.			Graduate Student
  3238 Bishop Street #2.	www.ececs.uc.edu/~alin
  Cincinnati. OH - 45220.	Email: alin@ececs.uc.edu
---------------------------------------------------------

Re: TLB miss handler code

[David S. Miller]

So, in nearly 3 days of this thread nobody has discovered
the arch/sparc64/kernel/{itlb,dtlb}_*.S files yet.

I find this particularly amusing :-)

Re: TLB miss handler code

[Ed L Cashin]

Nawab Ali <alin@ececs.uc.edu> writes:

> Hi Ed,
> Thanx for pointing out the file. I checked out the do_sparc64_fault
> function in arch/sparc64/mm/fault.c (2.6.0).
>
> This is the code snippet I found.
> if ((fault_code & FAULT_CODE_ITLB) && (fault_code & FAULT_CODE_DTLB))
> 	BUG();

That case is a bug, so it's not what you're looking for.  The test
checks whether the fault code says that it's a dTLB *and* an iTLB
miss, which doesn't make any sense.

You're interested in the non-buggy case.  In 2.6.0-test11, I can
follow that the call sequence: do_sparc64_fault calls
mm/memory.c:handle_mm_fault which calls handle_pte_fault.  Eventually
update_mmu_cache and the assembly __update_mmu_cache are called.

arch/sparc64/mm/ultra.S has the actual code that fills the TLBs.  All
you have to do is decide where in that path it makes sense for you to
add your code.  Maybe about the time update_mmu_cache is called.

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[Ed L Cashin]

"David S. Miller" <davem@redhat.com> writes:

> So, in nearly 3 days of this thread nobody has discovered
> the arch/sparc64/kernel/{itlb,dtlb}_*.S files yet.
>
> I find this particularly amusing :-)

It is funny, but I bet there are lurkers who know and aren't speaking
up.

Examining those files (although my sparc asm isn't good) it looks like
not all TLB misses go through do_sparc64_fault, which I suppose is
your point.  In fact, it looks like do_sparc64_fault gets invoked only
when the instruction causing the TLB miss is also causing a page
fault.  I had mistakenly assumed that (fault_code & FAULT_CODE_?TLB)
inside of do_sparc64_fault meant it was just a TLB miss.

Rather, if it's just a TLB miss, the fast handler itself takes care of
filling the TLB.  (I'm unfamiliar with the "retry" instruction, but
the comment says "Trap return".)  

I really do not understand how it is getting the pte so fast.  It
looks like it doesn't do any locking but just performs a couple quick
calculations to get the address of the pte.

Incidentally, why are those fast TLB-miss handlers in
arch/sparc64/kernel and not arch/sparc64/mm?

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[David S. Miller]

On Wed, 21 Jan 2004 12:21:06 -0500
Ed L Cashin <ecashin@uga.edu> wrote:

> I really do not understand how it is getting the pte so fast.  It
> looks like it doesn't do any locking but just performs a couple quick
> calculations to get the address of the pte.

The page tables of the current process are mapped virtually and linearlly
starting at VPTE_BASE, in this way a PTE lookup is merely computed via
VPTE_BASE + (tlb_miss_vaddr >> SHIFT).

This can, itself, cause a TLB miss, for the VPTE_BASE page table mapping,
which is serviced by the code in dtlb_backend.S.

> Incidentally, why are those fast TLB-miss handlers in
> arch/sparc64/kernel and not arch/sparc64/mm?

Bacause these handlers are included directly in the trap table.

Re: TLB miss handler code

[Ed L Cashin]

"David S. Miller" <davem@redhat.com> writes:

...
> The page tables of the current process are mapped virtually and linearlly
> starting at VPTE_BASE, in this way a PTE lookup is merely computed via
> VPTE_BASE + (tlb_miss_vaddr >> SHIFT).
>
> This can, itself, cause a TLB miss, for the VPTE_BASE page table mapping,
> which is serviced by the code in dtlb_backend.S.

That seems like a very clever idea.  At least it's very efficient!  

>> Incidentally, why are those fast TLB-miss handlers in
>> arch/sparc64/kernel and not arch/sparc64/mm?
>
> Bacause these handlers are included directly in the trap table.

OK.  Thanks much for the response.  It is a lot more concrete now.

I think Nawab Ali (the OP) has his work cut out for him, though.
Logging the pte on every TLB miss sounds tricky at best, given that
these fast handlers will service most misses.

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[Nawab Ali]

On Wed, 21 Jan 2004, Ed L Cashin wrote:

> "David S. Miller" <davem@redhat.com> writes:
>
> OK.  Thanks much for the response.  It is a lot more concrete now.
>
> I think Nawab Ali (the OP) has his work cut out for him, though.
> Logging the pte on every TLB miss sounds tricky at best, given that
> these fast handlers will service most misses.

Does it mean I'll have to muck around with the
arch/sparc64/kernel/{itlb,dtlb}_*.S files if I have to log(create a
tracefile) of all addresses that result in a TLB miss. My understanding
of sparc asm is not that great and I was wondering if there is any other way.

Thanx for all the help guys.

-nawab

> --
> --Ed L Cashin            |   PGP public key:
>   ecashin@uga.edu        |   http://noserose.net/e/pgp/
>
> -
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

==
Ask not what Mozilla can do for you...
---------------------------------------------------------
  Nawab Ali.			Graduate Student
  3238 Bishop Street #2.	www.ececs.uc.edu/~alin
  Cincinnati. OH - 45220.	Email: alin@ececs.uc.edu
---------------------------------------------------------

Re: TLB miss handler code

[David S. Miller]

On Wed, 21 Jan 2004 18:14:05 -0500 (EST)
Nawab Ali <alin@ececs.uc.edu> wrote:

> Does it mean I'll have to muck around with the
> arch/sparc64/kernel/{itlb,dtlb}_*.S files if I have to log(create a
> tracefile) of all addresses that result in a TLB miss.

That's exactly correct.

In fact, you must not even change the number of instructions in these
routines because they fit _PRECISELY_ into the sparc v9 trap table
slots they are in.  You would have to branch to some routine somewhere
else in the kernel in order to add even one extra instruction of code.

Re: TLB miss handler code

[Ed L Cashin]

"David S. Miller" <davem@redhat.com> writes:

> On Wed, 21 Jan 2004 18:14:05 -0500 (EST)
> Nawab Ali <alin@ececs.uc.edu> wrote:
>
>> Does it mean I'll have to muck around with the
>> arch/sparc64/kernel/{itlb,dtlb}_*.S files if I have to log(create a
>> tracefile) of all addresses that result in a TLB miss.
>
> That's exactly correct.
>
> In fact, you must not even change the number of instructions in these
> routines because they fit _PRECISELY_ into the sparc v9 trap table
> slots they are in.  You would have to branch to some routine somewhere
> else in the kernel in order to add even one extra instruction of code.

Is it just that simple?  Nawab Ali could, e.g., copy a section "S" of
the trap handler code to a different part of the kernel and replace S
with a jump to the new location.  It would be a lot slower, I
suppose. 

Then, with S in the regular kernel, he could add whatever he wanted,
including jumps to C functions that he writes himself.  That way he
could write his own stuff in C with minimal modifications to the asm.

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[David S. Miller]

On Wed, 21 Jan 2004 21:15:58 -0500
Ed L Cashin <ecashin@uga.edu> wrote:

> Is it just that simple?  Nawab Ali could, e.g., copy a section "S" of
> the trap handler code to a different part of the kernel and replace S
> with a jump to the new location.  It would be a lot slower, I
> suppose. 
> 
> Then, with S in the regular kernel, he could add whatever he wanted,
> including jumps to C functions that he writes himself.  That way he
> could write his own stuff in C with minimal modifications to the asm.

There are also very strict restrictions regarding register usage, you only
have 4 or 5 global registers to use, the other 2 or 3 global registers
have hardcoded values and furthermore if the VPTE_BASE mapping takes a TLB
miss the miss handler for that knows what values are precomputed in global
registers by the top-level TLB miss handler.

He cannot even touch any data structures as that would cause potential
recursive TLB misses and corrupt the current handler.

In short, I would only recommend this work to a true expert in Sparc v9
and UltraSPARC TLB programming.  I myself would take a few long days to
implement said tracing support.

Re: TLB miss handler code

[Ed L Cashin]

"David S. Miller" <davem@redhat.com> writes:

...
> There are also very strict restrictions regarding register usage, you only
> have 4 or 5 global registers to use, the other 2 or 3 global registers
> have hardcoded values and furthermore if the VPTE_BASE mapping takes a TLB
> miss the miss handler for that knows what values are precomputed in global
> registers by the top-level TLB miss handler.
>
> He cannot even touch any data structures as that would cause potential
> recursive TLB misses and corrupt the current handler.
>
> In short, I would only recommend this work to a true expert in Sparc v9
> and UltraSPARC TLB programming.  I myself would take a few long days to
> implement said tracing support.

Wow.  I had a feeling it might not be so simple.  ;)

Sounds like the only hope for Nawab Ali is if there's a simulator for
the sparc environment where he wants to perform the measuring.

I don't know anything about performance counters on ultrasparc, but
even if there are some, it wouldn't be enough to log the ptes that get
loaded on TLB misses.

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/

Re: TLB miss handler code

[Ed L Cashin]

"David S. Miller" <davem@redhat.com> writes:

...
> There are also very strict restrictions regarding register usage, you only
> have 4 or 5 global registers to use, the other 2 or 3 global registers
> have hardcoded values and furthermore if the VPTE_BASE mapping takes a TLB
> miss the miss handler for that knows what values are precomputed in global
> registers by the top-level TLB miss handler.
>
> He cannot even touch any data structures as that would cause potential
> recursive TLB misses and corrupt the current handler.
>
> In short, I would only recommend this work to a true expert in Sparc v9
> and UltraSPARC TLB programming.  I myself would take a few long days to
> implement said tracing support.

Wow.  I had a feeling it might not be so simple.  ;)

Sounds like the only hope for Nawab Ali is if there's a simulator for
the sparc environment where he wants to perform the measuring.

I don't know anything about performance counters on ultrasparc, but
even if there are some, it wouldn't be enough to log the ptes that get
loaded on TLB misses.

-- 
--Ed L Cashin            |   PGP public key:
  ecashin@uga.edu        |   http://noserose.net/e/pgp/