System suffers frequent TLB miss

System suffers frequent TLB miss

[wilbur.chan]

Hi all

We have a system running on mips64 xlr 732.  Our major application
process is binded on CPU5,

In order to reduce the tlb miss of our major process, we took the
following steps:

(I)  Use 2 number of  tlb entries  to map  the elf code segment and
data segment, say,  tlb_entry[i] { code segment}   and tlb_entry[i+1]
{data segment}  respectively.

(II)  Use another  6 number of tlb entries to map 6 reserved memory
regions. In this way, our process can manipulate these

   6 regions without any tlb miss.


However we found that,  the tlib miss frequency  for (I) and (II) is
very high.

We guess the reson for this is that, we use some malloc
operation,which leads to great tlb miss, and replace our tlb entries.

So we took messures to  isolate our tlb  entries and ordinary tlb
entres that were used for malloc.

(III)  In tlb_init function, we set the wried register to 6, so when
ordinary tlb miss occured, the tlb refill hander would write a random
tlb entry above 6,

 at the same time we can use our own 6 tlb entries to map whatever we want.




After this, we found that, process is still sufferring from TLB miss
in our 6 tlb entries.


I'm  totally exhausted about the tlb miss, I wonder if we can record
the virtual region of tlb miss  and the miss count in each region, in
that way,

I can find out which part leads to this tlb miss.That is , to record
C0_BADVADDR  in tlb miss.


However I'm not sure how to add code in build_r4000_tlb_refill_handler
function, for it is wrote in some strage way .


Any  suggestion on how to reduce tlb miss?

Thx in advance.

Re: System suffers frequent TLB miss

[wilbur.chan]

We record our 6 tlb missing  count in do_page_fault, because we don't
have page table mapping for these tlb entris,each time tlb miss

accured, tlb refilling handler would cause a page_fault

Re: System suffers frequent TLB miss

[Jayachandran C.]

On Sun, Apr 03, 2011 at 11:49:13AM +0800, wilbur.chan wrote:
> Hi all
> 
> We have a system running on mips64 xlr 732.  Our major application
[...]
> I'm  totally exhausted about the tlb miss, I wonder if we can record
> the virtual region of tlb miss  and the miss count in each region, in
> that way,
> 
> I can find out which part leads to this tlb miss.That is , to record
> C0_BADVADDR  in tlb miss.
> 
> 
> However I'm not sure how to add code in build_r4000_tlb_refill_handler
> function, for it is wrote in some strage way .
> 
> Any  suggestion on how to reduce tlb miss?

In our SDK linux, there is some code to add instructions in the TLB 
exception handler using the cpu scratch registers to count TLB misses.
(see mips/mm/tlbex.c code generated with OS_SCRATCH_REG2)

You can extend it with another scratch register containing the an address
per cpu where to record TLB miss badvaddr values(you would just need to 
add code to add sizeof(void*) to the scratch reg value and UASM_i_SW to
store addr). This should give you and idea which adresses are causing the
TLB miss.

I have not actually tried this out on linux(but I have implemented similar
code on FreeBSD and vxWorks) so it should work, let me know if you are
able to get this going...

JC.

Re: System suffers frequent TLB miss

[Ralf Baechle]

On Sun, Apr 03, 2011 at 11:49:13AM +0800, wilbur.chan wrote:

> We have a system running on mips64 xlr 732.  Our major application
> process is binded on CPU5,
> 
> In order to reduce the tlb miss of our major process, we took the
> following steps:

What page size are you using?  If you're still using 4K pages, try switching
to 16K or even 64K pages.  You also may want to look into using hugetlbfs.

  Ralf