LLC cache miss increases with the increases of hugepages

LLC cache miss increases with the increases of hugepages

[linhaifeng]

Hi，

I did a network forward test with ovs-dpdk, and found that LLC cache miss
increases with the increases of hugepages. ovs-dpdk only use 4 hugepages,
hugepage size is 1G.Why LLC cache miss increases ? I just echo more hugepages
but haven't been used yet.

some informations:

CPU: Intel(R) Xeon(R) CPU E5-2620 v3@ 2.40GHz
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K

echo 10 > /proc/sys/vm/nr_hugepages
ovs-dpdk's LLC-load-misses:3.88%
ovs-dpdk's iTLB-load-misses: 4.37%


echo 100 > /proc/sys/vm/nr_hugepages
reboot VM restart test.
ovs-dpdk's LLC-load-misses:31.55%
ovs-dpdk's iTLB-load-misses: 36.36%

I use perf to watch LLC-loads when not run test:

Samples: 1M of event 'LLC-loads', Event count (approx.): 57240206
Overhead  Shared Object                 Symbol
   9.19%  [kernel]                      [k] copy_page_rep
   5.03%  [kernel]                      [k] ktime_get
   4.06%  [unknown]                     [.] 0x000000000043e39d
   2.65%  bash                          [.] 0x000000000003e39d
   2.13%  [kernel]                      [k] rcu_check_callbacks

The function copy_page_rep seems like would prefetch all hugepages?