[RFC] dama: userspace DAMON_RECLAIM min_age autotuner

[Liew Rui Yan <aethernet65535@gmail.com>]

Hi DAMON Community,

I've been working on a userspace program that autotunes the 'min_age'
parameter of DAMON_RECLAIM, aiming to reduce unnecessary proactive
reclaim. I call it DAMA [1] (DAMOS Autotuner/Assistant).

Test Setup
==========

I wrote a synthetic test using Masim. The workload runs for 30 minutes
with two repeating scenarios:

Scenario 1
----------

- Full access (100% of region A) for 30s, then only 10% for 30s,
  repeating.
- Theoretical optimal min_age: > 30s.

Scenario 2
----------

- Cycle through four regions {A, B, C, D}, accessing 95% of each for 30s
  before moving to the next. A region is revisited every 90s.
- Theoretical optimal min_age: < 90s.

(Note: because DAMON default configuration does not partition regions
with high precision, the theoretical bounds are not strict in practice.)

Results
-------

I compared DAMA (starting min_age = 10s) against three configurations:

  - Default : DAMON_RECLAIM with 120s fixed min_age
  - Custom  : 60s fixed min_age
  - System  : no DAMON (kswapd + direct reclaim only)

All reclaim/refault counts are in pages. PSI values are averages. Fault
numbers are per-second rates.

    |-------------------------------------------------------------|
    |           | DEFAULT   | CUSTOM      | DAMA      | SYSTEM    |
    |-------------------------------------------------------------|
    | RECLAIMED | --------- | ----------- | --------- | --------- |
    | DAMON     | 0         | 27 648      | 669 274   | 0         |
    | KSWAPD    | 4 876 306 | 5 259 842   | 1 817 669 | 5 341 815 |
    | DIRECT    | 12 670    | 19 697      | 1 479     | 20 114    |
    | PSI       | --------- | ----------- | --------- | --------- |
    | CPU       | 0.06      | 0.06        | 0.06      | 0.06      |
    | I/O       | 0.00      | 0.00        | 0.00      | 0.00      |
    | MEM       | 0.22      | 0.23        | 0.08      | 0.23      |
    | REFAULT   | --------- | ----------- | --------- | --------- |
    | ANON      | 4 462 273 | 4 858 679   | 2 015 817 | 4 915 695 |
    | FAULT     | --------- | ----------- | --------- | --------- |
    | PGFAULT   | 1 317.48  | 1 428.33    | 575.10    | 1 443.96  |
    | MAJFAULT  | 1 239.70  | 1 349.85    | 560.25    | 1 365.70  |
    |-------------------------------------------------------------|

DAMA significantly reduces system reclaim, refaults and major faults
while keeping memory pressure low.

Masim Script
------------

In short, four regions share 8 GiB:
user_A_sysadmin (40%), user_B_student (20%), user_C_drive (20%),
user_D_bad (20%). The two access patterns above are looped to fill 30
minutes.

Here's the full Python script to generate Masim script:

    from masim_config import Region, AccessPattern, Phase, pr_config

    KiB = 1 * 1024
    MiB = 1024 * KiB
    GiB = 1024 * MiB

    SEC_MS = 1000
    MIN_MS = 60 * SEC_MS

    total_mem = 8 * GiB

    regions = [
        Region('user_A_sysadmin', int(total_mem * 0.4), 'none'),
        Region('user_B_student', int(total_mem * 0.2), 'none'),
        Region('user_C_drive', int(total_mem * 0.2), 'none'),
        Region('user_D_bad', int(total_mem * 0.2), 'none'),
    ]

    phases = []
    
    # Scenario 1
    # ==========
    #
    # Loop 30s full access + 30s partial access.
    #
    # Total time: 30mins ((30s + 30s) * 30 loops)
    #
    # The 'min_age' should not be too small, otherwise it will cause many
    # unnecessary proactive reclaim.
    #
    # Ideal 'min_age': > 30s

    for i in range(30):
        phases.append(Phase(f'scene1_high_{i}', 30 * SEC_MS, [
            AccessPattern('user_A_sysadmin', False, 1024, 100, 'rw'),
        ]))
        phases.append(Phase(f'scene1_low_{i}', 30 * SEC_MS, [
            AccessPattern('user_A_sysadmin', False, 1024, 10, 'rw'),
        ]))
    
    # Scenario 2
    # ==========
    #
    # Looping {A, B, C, D} with alternating 95% access.
    # Each region is re-access every 90s.
    #
    # Total time: 30mins ((30s * 4 times) *  15 loops)
    #
    # The 'min_age' should not be too large, otherwise it will cause many
    # system memory reclaim (kswapd/direct).
    #
    # Ideal 'min_age': < 90s

    region_names = ['user_A_sysadmin', 'user_B_student', 'user_C_drive', 'user_D_bad']
    for i in range(15):
        for r_name in region_names:
            phases.append(Phase(f'scene2_{r_name}_{i}', 30 * SEC_MS, [
                AccessPattern(r_name, True, 0, 95, 'rw'),
            ]))

    pr_config(regions, phases)

Algorithm
=========

DAMA's "DAMON_RECLAIM's min_age" algorithm is a periodic feedback
controller (core.c:reclaim_min_age_calc()) that balances DAMON_RECLAIM
and system reclaim to keep refaults low.

1. Accumulation & Decay
   Each cycle, DAMA reads the delta of DAMON-reclaimed pages, system
   reclaim (kswapd + direct), and refaults (anon + file). These deltas
   are added to two independent "remaining" counters:
   - damon_remaining  (for DAMON reclaim)
   - pgsteal_remaining (for kswapd + direct reclaim)

   Simultaneously, the same deltas are accumulated into long-lived
   metrics and continuously decayed by a fixed factor to smooth out
   short-term spikes.

2. Threshold Gating & Hysteresis
   An adjustment is only allowed when one of the remaining counters has
   built up enough "credit":
   - If damon_remaining >= DAMON_THRESHOLD and DAMON reclaimed pages in
     the current cycle, the controller considers _increasing_ min_age.
   - If pgsteal_remaining >= PGSTEAL_THRESHOLD and system reclaim is
     non-zero, it considers _decreasing_ min_age.

   Once a threshold is met, the _opposite_ remaining counter is rapidly
   decayed (multiplied by NOT_WORKING_FACTOR) to prevent the controller
   from oscillating between the two directions.

3. Decision
   - Increase min_age: compute (weighted_refault * 100) /
     damon_reclaimed. If this percentage exceeds INCREASE_THRESHOLD,
     DAMA assumes DAMON is evicting active pages and raises min_age
     proportionally.
   - Decrease min_age: compute (weighted_refault * 100) / (kswapd +
     direct reclaimed). If the percentage is below DECREASE_THRESHOLD,
     DAMA assumes system reclaim is missing cold pages and lowers
     min_age proportionally.

   After every adjustment, the refault counters are zeroed and the
   metrics continue to age, so the next decision is based on fresh,
   representative data.
   
Question
========

Synthetic tests have clear boundaries, so I'd appreciate your thoughts
on how to make the benchmark more representative of real production
workloads. What memory access patterns or workloads do you usually
consider when evaluating DAMOS self-adaptive capabilities?

I'd also love any feedback on this userspace autotuning approach or
suggestions for real-world testing.

[1] https://github.com/aethernet65535/dama

Best regards,
Rui Yan