Re: [RFC PATCH v6 0/5] mm: Hot page tracking and promotion infrastructure

[Bharata B Rao <bharata@amd.com>]

NAS Parallel Benchmarks - BT results

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
         in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
         use of hint faults as source in the pghot case.
         Both promotion and demotion are enabled in this case.
NUMAB3 - Enabled both regular and tiering mode of NUMA Balancing
         (kernel.numa_balancing=3)

Pghot by default promotes after two accesses but for NUMAB2 source,
promotion is done after one access to match the base behaviour.
(/sys/kernel/debug/pghot/freq_threshold=1)

NAS-BT details
--------------
Command: mpirun -np 16 /usr/bin/numactl --cpunodebind=0,1
NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the benefit
of promotion), class E results in around 368G of memory which overflows my
toptier. Hence I wanted something in between these classes. So I have  modified
class F to the problem size of 768 which results in around 160GB of memory.

After the memory consumption stabilizes, all the rank PIDs are paused and
their memory is moved to CXL node using migratepages command. This simulates
the situation of memory residing on lower tier node and access by BT processes
leading to promotion.

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
                        Base            Base            pghot-default
pghot-precise
                        NUMAB0          NUMAB2          NUMAB2          NUMAB2
=====================================================================================
Time in seconds         7321.79         4333.85         6498.78         4386.27
Mop/s total             53451.77        90303.780       60221.01        89224.51

pgpromote_success       0               41971151        423163051       41957809
pgpromote_candidate     0               0               1870949786      0
pgpromote_candidate_nrl 0               41971151        29360089        41957809
pgdemote_kswapd         0               0               391179763       0
numa_pte_updates        0               42041312        1919944389      2568923206
numa_hint_faults        0               41972330        1911683592      2562729196
=====================================================================================
                                                        pghot-default
                                                        NUMAB3
=====================================================================================
Time in seconds                                         4425.84
Mop/s total                                             88426.77

pgpromote_success                                       41957442
pgpromote_candidate                                     0
pgpromote_candidate_nrl                                 41957442
pgdemote_kswapd                                         0
numa_pte_updates                                        2588634775
numa_hint_faults                                        2581645889
=====================================================================================

- In the base case, the benchmark numbers improve significantly due to hot page
  promotion.
- Though the benchmark runs for hundreds of minutes, the pages get promoted
  within the first few mins.
- pghot-precise is able to match the base case numbers.
- The benchmark suffers in pghot-default case due to promotion being limited
  to the default NID (0) only. This leads to excessive PTE updates, hint faults,
  demotion and promotion churn.
- With NUMAB3, pghot-default case recovers the performce as in this mode
  misplaced hot page migrations get right placed due to NUMA balancing
  mode=1 being active.