Re: [PATCH RFC] mm: memory-tiering: Fix PGPROMOTE_CANDIDATE accounting

["Huang, Ying" <ying.huang@linux.alibaba.com>]

"Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com> writes:

> On 20/06/2025 14:28, Huang, Ying wrote:
>> Li Zhijian <lizhijian@fujitsu.com> writes:
>> 
>>> Goto-san reported confusing pgpromote statistics where
>>> the pgpromote_success count significantly exceeded pgpromote_candidate.
>>> The issue manifests under specific memory pressure conditions:
>>> when top-tier memory (DRAM) is exhausted by memhog and allocation begins
>>> in lower-tier memory (CXL). After terminating memhog, the stats show:
>> 
>> The above description is confusing.  The page promotion occurs when the
>> size of the top-tier free space is large enough (after killing the
>> memhog above).  The accessed lower-tier memory will be promoted upon
>> accessing to take full advantage of the more expensive top-tier memory.
>
> Yeah, that's what the promotion does.
>
> Let's clarify the reproducer steps specifically(thanks Goto-san for the reproducer):
> On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
>
> # Enable demotion only
> echo 1 > /sys/kernel/mm/numa/demotion_enabled
> numactl -m 0-1 memhog -r200 3500M >/dev/null &
> pid=$!
> sleep 2
> numactl memhog -r100 2500M >/dev/null &
> sleep 10
> kill -9 $pid
> # Enable promotion
> echo 2 > /proc/sys/kernel/numa_balancing
>
> # After a few seconds, we observe `pgpromote_candidate < pgpromote_success`
>
> In this scenario, after terminating the first memhog, the conditions
> for pgdat_free_space_enough() are quickly met, triggering promotion.
> However, these migrated pages are only accounted for in PGPROMOTE_SUCCESS, not in PGPROMOTE_CANDIDATE.

Yes.  This is the expected behavior of current implementation.

>
>> 
>>> $ grep -e pgpromote /proc/vmstat
>>> pgpromote_success 2579
>>> pgpromote_candidate 1
>>>
>>> This update increments PGPROMOTE_CANDIDATE within the free space branch
>>> when a promotion decision is made, which may alter the mechanism of the
>>> rate limit. Consequently, it becomes easier to reach the rate limit than
>>> it was previously.
>>>
>>> For example:
>>> Rate Limit = 100 pages/sec
>>> Scenario:
>>>    T0: 90 free-space migrations
>>>    T0+100ms: 20-page migration request
>>>
>>> Before:
>>>    Rate limit is *not* reached: 0 + 20 = 20 < 100
>>>    PGPROMOTE_CANDIDATE: 20
>>> After:
>>>    Rate limit is reached: 90 + 20 = 110 > 100
>>>    PGPROMOTE_CANDIDATE: 110
>> 
>> Yes.  The rate limit will be influenced by the change.  So, more tests
>> may be needed to verify it will not incurs regressions.
>
>
> Testing this might be challenging due to workload dependencies. Do you
> have any recommended workloads for evaluation?

Some in-memory database should be good workloads, for example, redis, etc.

> Alternatively, could we could rely on the LKP project for impact assessment(Current patch has not really tested
> by LKP due to a compiling error, I will post a V2 soon).

LKP has some basic workload to test this, for example, pmbench with
Gauss-ih access pattern.

> However, regarding the rate limit change itself, I consider this patch
> logically correct. As stated in the numa_promotion_rate_limit()
> comment:
>> "For memory tiering mode, too high promotion/demotion throughput may hurt application latency."
> It seems there is no justification for excluding
> pgdat_free_space_enough() triggered promotions from the rate limiting
> mechanism.

In fact, we don't rate limit promotion if there are enough free space on
fast memory to fill the fast memory quickly.  I think that it's
necessary to prevent the fast memory from under-utilized ASAP.

>
>
>> 
>>>
>>> Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
>>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>

[snip]

---
Best Regards,
Huang, Ying