Re: Memory tiering with DAMON/DAMOS auto-tuning

[SJ Park <sj@kernel.org>]

On Wed, 30 Sep 2026 07:03:47 +0300 Anton Gavriliuk <antosha20xx@gmail.com> wrote:

> > Oops...  Seems you were running next branch of damo.  There was a bug.  I
> > reproduced it on 7.2.8 kernel, and fixed it.  The fix [1] is now pushed.  Could
> > you pull the 'next' branch and try again?
> 
> Done.
[...]
> It looks now it keeps ~50% free numa node 0,

Awesome :D

[...]
> [root@localhost ~]# /home/anton/damo/damo report damon
> kdamond 0
>     state: on, pid: 29836
>     context 0
>         ops: paddr
[...]
>         scheme 0
>             action: migrate_cold to node 2 per 1 s
>             target access pattern
>                 sz: [0 B, max]
>                 nr_accesses: [0 samples, 0 samples]
>                 age: [0 aggr_intervals, 184,467,440,737,095 aggr_intervals]
>             quotas
>                 0 ns / 8.000 GiB / 0 B per 1 s
>                 goal 0: metric node_mem_free_bp (nid 0) target 5,000 current 0
>                 goal tuner: temporal

Ok, the above line confirms DAMON is running with 'temporal' tuner.

[...]
> DAMON & DAMOS are very flexible, so it requires a deep understanding
> of specific workload and how to configure memory tiering with DAMON &
> DAMOS.

Indeed it is.  Nonetheless, for the reason we provide DAMON modules [1] or damo
scripts for commonly known DAMON usages.  We have DAMON_RECLAIM module [2] for
proactive memory reclamation, and mem_tier.sh [3] for memory tiering.

My suggestion is to start from such examples, find what is not working and
asking questions to me.  So you are doing all great :D

> 
> So firstly I decided to make it like VMware VCF 9.1 does :-)), next I
> will reduce the goal from 50% free to 25-25% for numa node 0.

Sounds like a good plan.

> 
> Based on the outputs above, please let me know if other improvements
> are required.

This temporal tuner test has proved the quota system is not broken.  It implies
the previous test resulted in migrating nearly all data to the lower tier,
because there was no hot data to promote back to the upper tier.

In many common benchmarks using zipfian-like access distribution, hot data is
always hot, and cold data is always cold.  And usually the amount of cold data
is much larger than hot data.  I found [4] the pattern is making evaluation of
tiering solutions difficult, and was using artificial access pattern mixing to
work around.

My imagined real world workload is, there will be hot and cold data.  And the
pattern will nearly always be stable.  But, occasionally there will be changes.
Say, a chunk of data will be hot for a few hours, but suddenly be cold and keep
being cold for a few hours, then sudenly be warm for hours, and so on.  The
auto-tuning based DAMON tiering is designed with such workload in mind.

For your testing, I think the setup is good.  I'd suggest lower target free
memory ratio, though.  Assuming the theory (your workload has static access
pattern of small hot data) is true, using consist tuner should also be fine.

You will have more than expected data in lower tier, but if those are truly
cold, why would we bother?  If you still want strict upper tier utilization,
you could make the access pattern and filter condition less strict.  Ideally,
the access pattern and filter conditions should all go away, assuming DAMON can
find true hot and cold data.  And I believe that would be the case for
long-running real world workloads.  Short-running test workload would show
DAMON making wrong decisions in short term, though.

If you want to make sure if my theory is true, you could also profile the
access pattern of your test workload using DAMON.  I actually did it for my
auto-tuned tiering test [4] to find the fact.

[1] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#special-purpose-access-aware-kernel-modules
[2] https://origin.kernel.org/doc/html/latest/admin-guide/mm/damon/reclaim.html
[3] https://github.com/damonitor/damo/blob/next/scripts/mem_tier.sh
[4] https://lkml.kernel.org/r/20250420194030.75838-1-sj@kernel.org


Thanks,
SJ

[...]