Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure

[Bharata B Rao <bharata@amd.com>]

On 06-Aug-26 7:14 PM, SJ Park wrote:
> On Thu, 6 Aug 2026 11:19:22 +0530 Bharata B Rao <bharata@amd.com> wrote:
> 
>> On 28-Jul-26 11:44 PM, Andrew Morton wrote:
>>> On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <bharata@amd.com> wrote:
>>>
>>>> This patchset introduces pghot, a subsystem for hot page tracking and
>>>> promotion.
>>>
>>> Can DAMON be used to do this sort of thing adequately?
>>
>> Hi SJ,
>>
>> I started comparing hot page detection and promotion aspects of DAMON
>> and pghot through benchmark runs. Here is the first set of results from
>> pointer chase workload.
>>
>> Since I wasn't very familiar with DAMON tunables and settings, I let the
>> AI to chose some default followed by other combinations. The test harness
>> is developed and run by AI. Request you to go through this and let me know
>> you feedback about the combinations and configurations tried here. Based on
>> that I can adapt the settings for the future runs.
> 
> This sounds like you asked AI to do the comparison because you don't know
> DAMON.  And you still don't know DAMON, so you cannot really explain the
> results.  So asking me to review what your AI did and guide your AI.  Sigh.
> 
> Correct me if I'm wrong.  If I'm not wrong, no, I'm not an AI slop reviewer.  I
> will not review that until this is clarified.  I call it AI slop regardless of
> the output when it is something that generated and shared without full
> understanding from the human user.
> 
> If you want me to review something, please send me what you really understand.
> To compare/discuss DAMON, please learn it first.  Spend time and ask question.
> I will be more than happy to help that.

I was seeking feedback about the combinations and configurations that
were tried so that I can adapt the setting for future runs. I spent time
looking at the rich set of configuration possibilities [1] that DAMON
provides and experimented with something very basic to start with, followed
by a few changes in each step. Now I can see more meaningful numbers
from DAMON without the need to enable migrate_cold.

Metric notes
------------
- lat  : steady access latency, ns/access, mean of last 25% of samples
         (lower is better).  Baselines: base 337 ns, pghot 348 ns
         (within 3%, so absolute latency is broadly comparable here).
- node0: end-of-run per-process placement on node 0 (numastat -p).
- CPU  : cumulative cputime of the monitor/promoter thread over the
         240 s run, and its peak instantaneous %cpu (ps).  This is the
         concentrated cost only; see caveat 2 on pghot's distributed
         cost.
- promoted: bytes actually migrated (DAMON scheme sz_applied; pghot
         pgpromote_success).


DAMON configuration method (default vs manually set)
----------------------------------------------------
The aim was to start from DAMON's own sysfs defaults and set only what
a promote scheme needs, then change one thing at a time.

Left at the DAMON default in every run below:
    sample=5ms, aggr=100ms, update=60s, apply_interval=0 (=aggr),
    intervals_goal off, watermarks none, quota goals none, filters
    none.  min_nr_regions=10 (except the page-size runs).

Set manually in every run (the minimum for migrate_hot to act):
    operations=paddr; one target region = node 2 physical range;
    action=migrate_hot; target_nid=0; access pattern nr_accesses>=1
    (sz>=1 page, any age), i.e. "promote anything accessed at least
    once in the aggregation window".

Per-run additions (the single variable that changes each row):
    D0 default    : nothing extra (pure default + migrate_hot).
    D1 page-size  : min=max nr_regions = 67,108,864 (one region per
                    4 KiB page over the 256 GiB target).
    D2 +quota     : quota bytes=128 MiB, reset=1000 ms (~128 MB/s),
                    weights sz/acc/age = 0/1000/0 (hottest first).
    D3 fine+quota : max_nr_regions=100,000 (min default 10) + D2 quota.
    D4 pagesz+q   : page-size regions (as D1) + a quota (1 GiB/1000 ms)
                    + the same weights.

Table 1.  DAMON, base kernel (each row changes one variable)
------------------------------------------------------------
id  config             regions       quota   lat   node0  over   kdamond
                        (settled/pin) (MB/s)  (ns)  (GB)          CPU
--  -----------------  -------------  ------  ----  -----  -----  ---------
B0  baseline           - (no DAMON)   none    337   0.0    -      -
D0  default (coarse)   11             none    121   64.0  +300%  25s/10%
D1  page-size          64M pin        none    129   33.0  +106%  277s/100%
D2  default + quota    14             128     105   23.3   +46%  9s/3.7%
D3  fine + quota       14-38          128      88   23.6   +47%  14s/5.6%
D4  page-size + quota  64M pin        ~1000   296    1.0   -94%  270s/100%

B0 is the no-promotion control on the base kernel (numa_balancing=0,
no DAMON), and is the reference for the lat column.

The "over" column is measured against the 16 GiB HOT SET, not against
B0: over = (node0 - 16 GiB) / 16 GiB.  B0 promotes nothing, so its
node0 is 0.0 and its over is n/a.

Regions column:
- A plain number (D0/D2/D3) is the SETTLED count DAMON adapted to,
  measured via the damon_aggregated tracepoint.  These rows leave
  min_nr_regions at the default 10; D3 only raises the max_nr_regions
  ceiling to 100,000 (a cap, not a target), so DAMON splits/merges
  freely and settles at 14-38.
- "64M pin" (D1/D4) means min_nr_regions = max_nr_regions = 67,108,864
  (one 4 KiB region per page of the 256 GiB target).  Setting min=max
  PINS the count: DAMON cannot merge/split, so it is forced, not
  settled.  The actual count was unmeasurable (NA) because the pegged
  kdamond never completed an aggregation pass to emit the tracepoint.

Table 2.  pghot, pghot kernel
-----------------------------
id  config            thr  lat  node0  over  promoted  kmigrated
                           (ns) (GB)         (GiB)     CPU
--  ----------------  ---  ---  -----  ----  --------  ---------
P0  baseline          -    348   0.0    -      -       0s
P2  hint-fault (def)  2    124  16.0    0%   16.0      6s/0.7%
P1  hint-fault        1    112  16.0    0%   15.9      13s/1.1%

Notes on Table 2:
- numa_balancing=2, and pghot tunables with the default values
 (pghot_enabled_sources=1 (hint-fault source), pghot_target_nid=0
  and pghot_promote_rate_limit_MBps)
- Both thresholds place exactly the 16 GiB hot set (0% over):
    P2 pgpromote_success 4,194,304 = 16.0 GiB exactly;
    P1 pgpromote_success 4,157,158 (15.9 GiB);
- thr=1 converges faster (first interval already 136 ns; it promotes
  during the load phase) and is slightly quicker (112 vs 124 ns) at
  marginally higher kmigrated CPU.  thr=2 starts at 395 ns and
  converges over the run.

Caveats
-------
1. Two kernels: DAMON on base, pghot on pghot.  The baselines are
   within 3% (337 vs 348 ns), so absolute latency is broadly
   comparable, but this is not a single-kernel A/B.
2. The CPU columns are the concentrated monitor/promoter thread only.
   pghot's hint-fault source also imposes distributed overhead on the
   application threads (NUMA scanning + hint-fault handling:
   numa_pte_updates 10-15M, numa_hint_faults 3-8M), which is NOT in the
   kmigrated figure.  DAMON's cost is concentrated in kdamond and is
   captured.  A fair total-cost comparison must add pghot's NUMA-
   balancing tax.
3. Latency does not track placement linearly for this workload and the
   mechanism was not isolated; placement/over-promotion is the primary
   accuracy metric.
4. DAMON and pghot increment different counters (scheme sz_applied vs
   pgpromote_success); cross-mechanism promotion counts are not
   directly comparable, so placement (numastat) is used.

[1] DAMON config directory - tree view
--------------------------------------
root@host:/sys/kernel/mm/damon/admin# tree
.
└── kdamonds
    ├── 0
    │   ├── contexts
    │   │   ├── 0
    │   │   │   ├── addr_unit
    │   │   │   ├── avail_operations
    │   │   │   ├── monitoring_attrs
    │   │   │   │   ├── intervals
    │   │   │   │   │   ├── aggr_us
    │   │   │   │   │   ├── intervals_goal
    │   │   │   │   │   │   ├── access_bp
    │   │   │   │   │   │   ├── aggrs
    │   │   │   │   │   │   ├── max_sample_us
    │   │   │   │   │   │   └── min_sample_us
    │   │   │   │   │   ├── sample_us
    │   │   │   │   │   └── update_us
    │   │   │   │   └── nr_regions
    │   │   │   │       ├── max
    │   │   │   │       └── min
    │   │   │   ├── operations
    │   │   │   ├── schemes
    │   │   │   │   ├── 0
    │   │   │   │   │   ├── access_pattern
    │   │   │   │   │   │   ├── age
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   ├── nr_accesses
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   └── sz
    │   │   │   │   │   │       ├── max
    │   │   │   │   │   │       └── min
    │   │   │   │   │   ├── action
    │   │   │   │   │   ├── apply_interval_us
    │   │   │   │   │   ├── core_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── dests
    │   │   │   │   │   │   └── nr_dests
    │   │   │   │   │   ├── filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── ops_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── quotas
    │   │   │   │   │   │   ├── bytes
    │   │   │   │   │   │   ├── effective_bytes
    │   │   │   │   │   │   ├── goals
    │   │   │   │   │   │   │   └── nr_goals
    │   │   │   │   │   │   ├── goal_tuner
    │   │   │   │   │   │   ├── ms
    │   │   │   │   │   │   ├── reset_interval_ms
    │   │   │   │   │   │   └── weights
    │   │   │   │   │   │       ├── age_permil
    │   │   │   │   │   │       ├── nr_accesses_permil
    │   │   │   │   │   │       └── sz_permil
    │   │   │   │   │   ├── stats
    │   │   │   │   │   │   ├── max_nr_snapshots
    │   │   │   │   │   │   ├── nr_applied
    │   │   │   │   │   │   ├── nr_snapshots
    │   │   │   │   │   │   ├── nr_tried
    │   │   │   │   │   │   ├── qt_exceeds
    │   │   │   │   │   │   ├── sz_applied
    │   │   │   │   │   │   ├── sz_ops_filter_passed
    │   │   │   │   │   │   └── sz_tried
    │   │   │   │   │   ├── target_nid
    │   │   │   │   │   ├── tried_regions
    │   │   │   │   │   │   └── total_bytes
    │   │   │   │   │   └── watermarks
    │   │   │   │   │       ├── high
    │   │   │   │   │       ├── interval_us
    │   │   │   │   │       ├── low
    │   │   │   │   │       ├── metric
    │   │   │   │   │       └── mid
    │   │   │   │   └── nr_schemes
    │   │   │   └── targets
    │   │   │       ├── 0
    │   │   │       │   ├── obsolete_target
    │   │   │       │   ├── pid_target
    │   │   │       │   └── regions
    │   │   │       │       ├── 0
    │   │   │       │       │   ├── end
    │   │   │       │       │   └── start
    │   │   │       │       └── nr_regions
    │   │   │       └── nr_targets
    │   │   └── nr_contexts
    │   ├── pid
    │   ├── refresh_ms
    │   └── state
    └── nr_kdamonds

29 directories, 60 files