Re: Memory tiering with DAMON/DAMOS auto-tuning

[SJ Park <sj@kernel.org>]

On Tue, 29 Sep 2026 11:41:15 +0300 Anton Gavriliuk <antosha20xx@gmail.com> wrote:

> Hello
> 
> I decided to move step-by-step, so firstly I would get on Linux/bare
> metal or non-VmWare setup memory tiering behaviour like on VmWare 9.1
> - https://knowledge.broadcom.com/external/article/449016/understanding-nvme-memory-tiering-activa.html#:~:text=Cause.%20This%20behavior%20is%20by%20design.%20The,page%20activity%20based%20on%20recency%20and%20frequency.
> 
> "Threshold Activation: The VMkernel initiates memory tiering only when
> total host physical DRAM (Tier 0) consumption reaches approximately
> the 80% threshold. If consumption is below this trigger (e.g., at
> 70-75%), the system will not migrate pages to Tier 1.
> Page Classification: ESXi monitors page activity based on recency and
> frequency. Only memory pages strictly classified as "cold" (inactive)
> are migrated to the NVMe Tier 1 device. The active working set ("hot"
> pages) is deliberately retained in Tier 0 to prevent performance
> penalties.
> Scan Rate: The page scan and migration rate scale with memory
> pressure. At lower consumption levels (just crossing the 80% mark),
> the rate is highly conservative."

Sounds good.

> 
> Here is my corrected command,
> 
> damo start \--numa_node 0 --monitoring_intervals_goal 5% 3 5ms 10s

Fyi, '--monitoring_intervals_autotune' is same to
'--monitoring_intervals_goal 4% 3 5ms 10s'.

> \--damos_action migrate_cold 2 --damos_access_rate 0% 0%
> \--damos_apply_interval 1s \--damos_quota_interval 1s
> --damos_quota_space 8GB \--damos_quota_goal node_mem_free_bp 50% 0
> \--damos_filter reject young \--numa_node 2
> --monitoring_intervals_goal 5% 3 5ms 10s \--damos_action migrate_hot 0
> --damos_access_rate 5% max \--damos_apply_interval 1s
> \--damos_quota_interval 1s --damos_quota_space 8GB \--damos_quota_goal
> node_mem_used_bp 50.3% 0 \--damos_filter allow young
> \--damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \--nr_targets 1 1
> --nr_schemes 1 1 --nr_ctxs 1 1

Also, fyi again, from damo v3.4.1, you can mark start of new kdamond, context,
target and scheme parameters on the command line using --kdamond, --damon_ctx,
--damon_target, and --damos_scheme.  you coud use those instead of
--damos_nr_quota_goals, --damos_nr_filters, --nr_targets, --nr_schemes and
--nr_ctxs.  E.g.,

damo start \
        ` # A kdamond to demote cold memory from node 0 to node 2 ` \
        --kdamond --damon_ctx --monitoring_intervals_autotune \
                --damon_target --numa_node 0 \
                --damos_scheme \
                        --damos_action migrate_cold 2 \
                        --damos_access_rate 0% 0% --damos_apply_interval 1s \
                        ` # up to 8 GiB per second ` \
                        --damos_quota_interval 1s --damos_quota_space 8G \
                        ` # aiming at least 50% node 0 free memory ` \
                        --damos_quota_goal node_mem_free_bp 50% 0 \
                        --damos_filter reject young \
        ` # A kdamond to promote hot memory from node 2 to node 0 ` \
        --kdamond --damon_ctx --monitoring_intervals_autotune \
                --damon_target --numa_node 2 \
                --damos_scheme \
                        --damos_action migrate_hot 0 \
                        --damos_access_rate 5% max --damos_apply_interval 1s \
                        ` # up to 8 GiB per second ` \
                        --damos_quota_interval 1s --damos_quota_space 8G \
                        ` # aiming at least 50.3% node 0 memory utilization ` \
                        --damos_quota_goal node_mem_used_bp 50.3% 0 \
                        --damos_filter allow young \

> 
> with consist -> temporal,
> 
> [root@localhost ~]# cat
> /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/schemes/0/quotas/goal_tuner
> temporal
> [root@localhost ~]# cat
> /sys/kernel/mm/damon/admin/kdamonds/1/contexts/0/schemes/0/quotas/goal_tuner
> temporal

Seems you manually made this change.  Has this executed after 'damo start'?
Also, did you 'commit' the updated commit input?  If any of your answer to the
questions is not "yes", the tuner update may not applied.

You can specify what tuner to use on 'damo' command together, using
'--damos_quota_goal_tuner' option.  I'd recommend using that.  E.g.,

damo start \
        ` # A kdamond to demote cold memory from node 0 to node 2 ` \
        --kdamond --damon_ctx --monitoring_intervals_autotune \
                --damon_target --numa_node 0 \
                --damos_scheme \
                        --damos_action migrate_cold 2 \
                        --damos_access_rate 0% 0% --damos_apply_interval 1s \
                        ` # up to 8 GiB per second ` \
                        --damos_quota_interval 1s --damos_quota_space 8G \
                        ` # aiming at least 50% node 0 free memory ` \
                        --damos_quota_goal node_mem_free_bp 50% 0 \
                        ` # using temporal tuner ` \
                        --damos_quota_goal_tuner temporal \
                        --damos_filter reject young \
        ` # A kdamond to promote hot memory from node 2 to node 0 ` \
        --kdamond --damon_ctx --monitoring_intervals_autotune \
                --damon_target --numa_node 2 \
                --damos_scheme \
                        --damos_action migrate_hot 0 \
                        --damos_access_rate 5% max --damos_apply_interval 1s \
                        ` # up to 8 GiB per second ` \
                        --damos_quota_interval 1s --damos_quota_space 8G \
                        ` # aiming at least 50.3% node 0 memory utilization ` \
                        --damos_quota_goal node_mem_used_bp 50.3% 0 \
                        ` # using temporal tuner ` \
                        --damos_quota_goal_tuner temporal \
                        --damos_filter allow young \

> 
> But it again didn't stop ~50%
> 
> [root@localhost ~]# numastat -p $(pgrep valkey-server)
> 
> Per-node process memory usage (in MBs) for PID 18036 (valkey-server)
>                            Node 0          Node 1          Node 2
>                   --------------- --------------- ---------------
> Huge                         0.00            0.00            0.00
> Heap                         0.11            0.00            0.00
> Stack                        0.03            0.00            0.00
> Private                 228807.54            0.48          185.79
> ----------------  --------------- --------------- ---------------
> Total                   228807.67            0.48          185.79
> 
>                            Node 3           Total
>                   --------------- ---------------
> Huge                         0.00            0.00
> Heap                         0.00            0.11
> Stack                        0.00            0.03
> Private                      0.00       228993.80
> ----------------  --------------- ---------------
> Total                        0.00       228993.94
> 
> 
> [root@localhost ~]# numastat -p $(pgrep valkey-server)
> 
> Per-node process memory usage (in MBs) for PID 18036 (valkey-server)
>                            Node 0          Node 1          Node 2
>                   --------------- --------------- ---------------
> Huge                         0.00            0.00            0.00
> Heap                         0.01            0.00            0.10
> Stack                        0.01            0.00            0.02
> Private                  13434.01            0.48       215559.31
> ----------------  --------------- --------------- ---------------
> Total                    13434.03            0.48       215559.43
> 
>                            Node 3           Total
>                   --------------- ---------------
> Huge                         0.00            0.00
> Heap                         0.00            0.11
> Stack                        0.00            0.03
> Private                      0.00       228993.80
> ----------------  --------------- ---------------
> Total                        0.00       228993.94

Interesting.  I doubt if the tuner change is correctly made.  'damo report
damon' can help us understand under what configuration DAMON is running.  It
could help us quickly see if the configuration is done as we intended.  Could
you share 'damo report damon' result on the final state?

It would also be helpful if you could run 'damo report damon --damos_stats'
periodically (say, once per 5-10 seconds) while the migration is ongoing and
share the outputs with us.

> 
> 
> 
> > > Oohhh... I moved to 5%.
> > I understand you mean 4%?
> 
> As far as I understood, this value represents "accuracy", so there
> shouldn't be a big difference between 4% vs 5%.  Or please correct me
> if I'm wrong.

Yes, 5% vs 4% should be an ignorable small difference.


Thanks,
SJ

[...]