> 
> Jack Steiner is trying to determine how much more expensive the off-node
> page tables are than on node.  Once we know that, we will know if a
> per-cpu or per-node quicklist type arrangement is truely beneficial or
> if a general use slab without node awareness will be adequate.
> 
> Thanks,
> Robin




Here is the results of a test that measures the benefit of
node-local page tables vs page tables located on a remote
node. This was run on a 64p Altix system using 900MHz/1.5M L3
cpus.

The test is very likely a ~worse-case test. It was contrived to 
maximize the number of TLB misses to page table entries that
are not in the cache of the processor. The test references
memory with a 32M+128 byte stride. The maximizes data cache
hits & minimizes PT cache hits. The page table pages were all 
allocated on 256K boundaries to ensure maximum cache conflicts
for PT references by the VHPT.

Granted, this is a worse case test but I wanted to see
if I could measure the effect. Although this test is atypical of
most apps, it is similar to some very large pointer chasing
applications in the real world.

The test essentially follows a linked list of pointers. When the number of 
entries in the list exceed the size of the TLB, TLB misses occur & 
the processor must make references to the in-memory page table.

      Memory Reference Time (ns/reference) for Pointer Chasing Test

 POINTERS    PT_LOCAL    PT_REMOTE
       1        5.621        5.621
      10        4.226        4.226
      20        3.224        3.224
      30        2.890        2.890
      40       11.418       11.418
      50       11.357       11.357
      60       11.316       11.316
      70       11.287       11.286
      80       11.329       11.302
      90       11.292       12.082
     100       11.272       12.763
     110       11.974       16.075
     120       51.758      132.389  << exceeds TLB capacity here
     130      119.311      382.279
     140      143.726      466.469
     150      158.851      495.946
     160      164.415      515.392
     170      168.822      525.382
     180      168.057      537.250
     190      173.043      536.515
     300      192.804      632.459
     400      204.916      666.561
     500      230.489      693.896
     600      256.716      725.338
     700      286.068      731.309
     800      293.407      740.035
     900      306.296      747.635


I've also attach a graph. It is prettier but may not be friendly
to all mail readers.
-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.